Add training configs for SuperPoint (MagicLeap) and DISK with LightGlue (#18)
* add superpoint and disk train configs * update READMEmain
parent
398c4b8c21
commit
692c72f94c
18
README.md
18
README.md
|
@ -25,7 +25,7 @@ python3 -m pip install -e .[extra]
|
||||||
All models and datasets in gluefactory have auto-downloaders, so you can get started right away!
|
All models and datasets in gluefactory have auto-downloaders, so you can get started right away!
|
||||||
|
|
||||||
## License
|
## License
|
||||||
The code and trained models in Glue Factory are released with an Apache-2.0 license. This includes LightGlue trained with an [open version of SuperPoint](https://github.com/rpautrat/SuperPoint). Third-party models that are not compatible with this license, such as SuperPoint (original) and SuperGlue, are provided in `gluefactory_nonfree`, where each model might follow its own, restrictive license.
|
The code and trained models in Glue Factory are released with an Apache-2.0 license. This includes LightGlue and an [open version of SuperPoint](https://github.com/rpautrat/SuperPoint). Third-party models that are not compatible with this license, such as SuperPoint (original) and SuperGlue, are provided in `gluefactory_nonfree`, where each model might follow its own, restrictive license.
|
||||||
|
|
||||||
## Evaluation
|
## Evaluation
|
||||||
|
|
||||||
|
@ -223,18 +223,18 @@ All training commands automatically download the datasets.
|
||||||
<details>
|
<details>
|
||||||
<summary>[Training LightGlue]</summary>
|
<summary>[Training LightGlue]</summary>
|
||||||
|
|
||||||
We show how to train LightGlue with [SuperPoint open](https://github.com/rpautrat/SuperPoint).
|
We show how to train LightGlue with [SuperPoint](https://github.com/magicleap/SuperPointPretrainedNetwork).
|
||||||
We first pre-train LightGlue on the homography dataset:
|
We first pre-train LightGlue on the homography dataset:
|
||||||
```bash
|
```bash
|
||||||
python -m gluefactory.train sp+lg_homography \ # experiment name
|
python -m gluefactory.train sp+lg_homography \ # experiment name
|
||||||
--conf gluefactory/configs/superpoint-open+lightglue_homography.yaml
|
--conf gluefactory/configs/superpoint+lightglue_homography.yaml
|
||||||
```
|
```
|
||||||
Feel free to use any other experiment name. By default the checkpoints are written to `outputs/training/`. The default batch size of 128 corresponds to the results reported in the paper and requires 2x 3090 GPUs with 24GB of VRAM each as well as PyTorch >= 2.0 (FlashAttention).
|
Feel free to use any other experiment name. By default the checkpoints are written to `outputs/training/`. The default batch size of 128 corresponds to the results reported in the paper and requires 2x 3090 GPUs with 24GB of VRAM each as well as PyTorch >= 2.0 (FlashAttention).
|
||||||
Configurations are managed by [OmegaConf](https://omegaconf.readthedocs.io/) so any entry can be overridden from the command line.
|
Configurations are managed by [OmegaConf](https://omegaconf.readthedocs.io/) so any entry can be overridden from the command line.
|
||||||
If you have PyTorch < 2.0 or weaker GPUs, you may thus need to reduce the batch size via:
|
If you have PyTorch < 2.0 or weaker GPUs, you may thus need to reduce the batch size via:
|
||||||
```bash
|
```bash
|
||||||
python -m gluefactory.train sp+lg_homography \
|
python -m gluefactory.train sp+lg_homography \
|
||||||
--conf gluefactory/configs/superpoint-open+lightglue_homography.yaml \
|
--conf gluefactory/configs/superpoint+lightglue_homography.yaml \
|
||||||
data.batch_size=32 # for 1x 1080 GPU
|
data.batch_size=32 # for 1x 1080 GPU
|
||||||
```
|
```
|
||||||
Be aware that this can impact the overall performance. You might need to adjust the learning rate accordingly.
|
Be aware that this can impact the overall performance. You might need to adjust the learning rate accordingly.
|
||||||
|
@ -242,17 +242,17 @@ Be aware that this can impact the overall performance. You might need to adjust
|
||||||
We then fine-tune the model on the MegaDepth dataset:
|
We then fine-tune the model on the MegaDepth dataset:
|
||||||
```bash
|
```bash
|
||||||
python -m gluefactory.train sp+lg_megadepth \
|
python -m gluefactory.train sp+lg_megadepth \
|
||||||
--conf gluefactory/configs/superpoint-open+lightglue_megadepth.yaml \
|
--conf gluefactory/configs/superpoint+lightglue_megadepth.yaml \
|
||||||
train.load_experiment=sp+lg_homography
|
train.load_experiment=sp+lg_homography
|
||||||
```
|
```
|
||||||
|
|
||||||
Here the default batch size is 32. To speed up training on MegaDepth, we suggest to cache the local features before training (requires around 150 GB of disk space):
|
Here the default batch size is 32. To speed up training on MegaDepth, we suggest to cache the local features before training (requires around 150 GB of disk space):
|
||||||
```bash
|
```bash
|
||||||
# extract features
|
# extract features
|
||||||
python -m gluefactory.scripts.export_megadepth --method sp_open --num_workers 8
|
python -m gluefactory.scripts.export_megadepth --method sp --num_workers 8
|
||||||
# run training with cached features
|
# run training with cached features
|
||||||
python -m gluefactory.train sp+lg_megadepth \
|
python -m gluefactory.train sp+lg_megadepth \
|
||||||
--conf gluefactory/configs/superpoint-open+lightglue_megadepth.yaml \
|
--conf gluefactory/configs/superpoint+lightglue_megadepth.yaml \
|
||||||
train.load_experiment=sp+lg_homography \
|
train.load_experiment=sp+lg_homography \
|
||||||
data.load_features.do=True
|
data.load_features.do=True
|
||||||
```
|
```
|
||||||
|
@ -297,10 +297,10 @@ Using the following local feature extractors:
|
||||||
| Model | LightGlue config |
|
| Model | LightGlue config |
|
||||||
| --------- | --------- |
|
| --------- | --------- |
|
||||||
| [SuperPoint (open)](https://github.com/rpautrat/SuperPoint) | `superpoint-open+lightglue_{homography,megadepth}.yaml` |
|
| [SuperPoint (open)](https://github.com/rpautrat/SuperPoint) | `superpoint-open+lightglue_{homography,megadepth}.yaml` |
|
||||||
| [SuperPoint (official)](https://github.com/magicleap/SuperPointPretrainedNetwork) | ❌ TODO |
|
| [SuperPoint (official)](https://github.com/magicleap/SuperPointPretrainedNetwork) | `superpoint+lightglue_{homography,megadepth}.yaml` |
|
||||||
| SIFT (via [pycolmap](https://github.com/colmap/pycolmap)) | `sift+lightglue_{homography,megadepth}.yaml` |
|
| SIFT (via [pycolmap](https://github.com/colmap/pycolmap)) | `sift+lightglue_{homography,megadepth}.yaml` |
|
||||||
| [ALIKED](https://github.com/Shiaoming/ALIKED) | `aliked+lightglue_{homography,megadepth}.yaml` |
|
| [ALIKED](https://github.com/Shiaoming/ALIKED) | `aliked+lightglue_{homography,megadepth}.yaml` |
|
||||||
| [DISK](https://github.com/cvlab-epfl/disk) | ❌ TODO |
|
| [DISK](https://github.com/cvlab-epfl/disk) | `disk+lightglue_{homography,megadepth}.yaml` |
|
||||||
| Key.Net + HardNet | ❌ TODO |
|
| Key.Net + HardNet | ❌ TODO |
|
||||||
|
|
||||||
## Coming soon
|
## Coming soon
|
||||||
|
|
|
@ -0,0 +1,47 @@
|
||||||
|
data:
|
||||||
|
name: homographies
|
||||||
|
data_dir: revisitop1m
|
||||||
|
train_size: 150000
|
||||||
|
val_size: 2000
|
||||||
|
batch_size: 128
|
||||||
|
num_workers: 14
|
||||||
|
homography:
|
||||||
|
difficulty: 0.7
|
||||||
|
max_angle: 45
|
||||||
|
photometric:
|
||||||
|
name: lg
|
||||||
|
model:
|
||||||
|
name: two_view_pipeline
|
||||||
|
extractor:
|
||||||
|
name: extractors.disk_kornia
|
||||||
|
max_num_keypoints: 512
|
||||||
|
force_num_keypoints: True
|
||||||
|
detection_threshold: 0.0
|
||||||
|
trainable: False
|
||||||
|
ground_truth:
|
||||||
|
name: matchers.homography_matcher
|
||||||
|
th_positive: 3
|
||||||
|
th_negative: 3
|
||||||
|
matcher:
|
||||||
|
name: matchers.lightglue
|
||||||
|
filter_threshold: 0.1
|
||||||
|
input_dim: 128
|
||||||
|
flash: false
|
||||||
|
checkpointed: true
|
||||||
|
train:
|
||||||
|
seed: 0
|
||||||
|
epochs: 40
|
||||||
|
log_every_iter: 100
|
||||||
|
eval_every_iter: 500
|
||||||
|
lr: 1e-4
|
||||||
|
lr_schedule:
|
||||||
|
start: 20
|
||||||
|
type: exp
|
||||||
|
on_epoch: true
|
||||||
|
exp_div_10: 10
|
||||||
|
plot: [5, 'gluefactory.visualization.visualize_batch.make_match_figures']
|
||||||
|
benchmarks:
|
||||||
|
hpatches:
|
||||||
|
eval:
|
||||||
|
estimator: opencv
|
||||||
|
ransac_th: 0.5
|
|
@ -0,0 +1,70 @@
|
||||||
|
data:
|
||||||
|
name: megadepth
|
||||||
|
preprocessing:
|
||||||
|
resize: 1024
|
||||||
|
side: long
|
||||||
|
square_pad: True
|
||||||
|
train_split: train_scenes_clean.txt
|
||||||
|
train_num_per_scene: 300
|
||||||
|
val_split: valid_scenes_clean.txt
|
||||||
|
val_pairs: valid_pairs.txt
|
||||||
|
min_overlap: 0.1
|
||||||
|
max_overlap: 0.7
|
||||||
|
num_overlap_bins: 3
|
||||||
|
read_depth: true
|
||||||
|
read_image: true
|
||||||
|
batch_size: 32
|
||||||
|
num_workers: 14
|
||||||
|
load_features:
|
||||||
|
do: false # enable this if you have cached predictions
|
||||||
|
path: exports/megadepth-undist-depth-r1024_DISK-k2048-nms5/{scene}.h5
|
||||||
|
padding_length: 2048
|
||||||
|
padding_fn: pad_local_features
|
||||||
|
model:
|
||||||
|
name: two_view_pipeline
|
||||||
|
extractor:
|
||||||
|
name: extractors.disk_kornia
|
||||||
|
max_num_keypoints: 512
|
||||||
|
force_num_keypoints: True
|
||||||
|
detection_threshold: 0.0
|
||||||
|
trainable: False
|
||||||
|
ground_truth:
|
||||||
|
name: matchers.homography_matcher
|
||||||
|
th_positive: 3
|
||||||
|
th_negative: 3
|
||||||
|
matcher:
|
||||||
|
name: matchers.lightglue
|
||||||
|
filter_threshold: 0.1
|
||||||
|
input_dim: 128
|
||||||
|
flash: false
|
||||||
|
checkpointed: true
|
||||||
|
allow_no_extract: True
|
||||||
|
train:
|
||||||
|
seed: 0
|
||||||
|
epochs: 50
|
||||||
|
log_every_iter: 100
|
||||||
|
eval_every_iter: 1000
|
||||||
|
lr: 1e-4
|
||||||
|
lr_schedule:
|
||||||
|
start: 30
|
||||||
|
type: exp
|
||||||
|
on_epoch: true
|
||||||
|
exp_div_10: 10
|
||||||
|
dataset_callback_fn: sample_new_items
|
||||||
|
plot: [5, 'gluefactory.visualization.visualize_batch.make_match_figures']
|
||||||
|
benchmarks:
|
||||||
|
megadepth1500:
|
||||||
|
data:
|
||||||
|
preprocessing:
|
||||||
|
side: long
|
||||||
|
resize: 1024
|
||||||
|
eval:
|
||||||
|
estimator: opencv
|
||||||
|
ransac_th: 0.5
|
||||||
|
hpatches:
|
||||||
|
eval:
|
||||||
|
estimator: opencv
|
||||||
|
ransac_th: 0.5
|
||||||
|
model:
|
||||||
|
extractor:
|
||||||
|
max_num_keypoints: 1024
|
|
@ -0,0 +1,47 @@
|
||||||
|
data:
|
||||||
|
name: homographies
|
||||||
|
data_dir: revisitop1m
|
||||||
|
train_size: 150000
|
||||||
|
val_size: 2000
|
||||||
|
batch_size: 128
|
||||||
|
num_workers: 14
|
||||||
|
homography:
|
||||||
|
difficulty: 0.7
|
||||||
|
max_angle: 45
|
||||||
|
photometric:
|
||||||
|
name: lg
|
||||||
|
model:
|
||||||
|
name: two_view_pipeline
|
||||||
|
extractor:
|
||||||
|
name: gluefactory_nonfree.superpoint
|
||||||
|
max_num_keypoints: 512
|
||||||
|
force_num_keypoints: True
|
||||||
|
detection_threshold: 0.0
|
||||||
|
nms_radius: 3
|
||||||
|
trainable: False
|
||||||
|
ground_truth:
|
||||||
|
name: matchers.homography_matcher
|
||||||
|
th_positive: 3
|
||||||
|
th_negative: 3
|
||||||
|
matcher:
|
||||||
|
name: matchers.lightglue
|
||||||
|
filter_threshold: 0.1
|
||||||
|
flash: false
|
||||||
|
checkpointed: true
|
||||||
|
train:
|
||||||
|
seed: 0
|
||||||
|
epochs: 40
|
||||||
|
log_every_iter: 100
|
||||||
|
eval_every_iter: 500
|
||||||
|
lr: 1e-4
|
||||||
|
lr_schedule:
|
||||||
|
start: 20
|
||||||
|
type: exp
|
||||||
|
on_epoch: true
|
||||||
|
exp_div_10: 10
|
||||||
|
plot: [5, 'gluefactory.visualization.visualize_batch.make_match_figures']
|
||||||
|
benchmarks:
|
||||||
|
hpatches:
|
||||||
|
eval:
|
||||||
|
estimator: opencv
|
||||||
|
ransac_th: 0.5
|
|
@ -0,0 +1,71 @@
|
||||||
|
data:
|
||||||
|
name: megadepth
|
||||||
|
preprocessing:
|
||||||
|
resize: 1024
|
||||||
|
side: long
|
||||||
|
square_pad: True
|
||||||
|
train_split: train_scenes_clean.txt
|
||||||
|
train_num_per_scene: 300
|
||||||
|
val_split: valid_scenes_clean.txt
|
||||||
|
val_pairs: valid_pairs.txt
|
||||||
|
min_overlap: 0.1
|
||||||
|
max_overlap: 0.7
|
||||||
|
num_overlap_bins: 3
|
||||||
|
read_depth: true
|
||||||
|
read_image: true
|
||||||
|
batch_size: 32
|
||||||
|
num_workers: 14
|
||||||
|
load_features:
|
||||||
|
do: false # enable this if you have cached predictions
|
||||||
|
path: exports/megadepth-undist-depth-r1024_SP-k2048-nms3/{scene}.h5
|
||||||
|
padding_length: 2048
|
||||||
|
padding_fn: pad_local_features
|
||||||
|
model:
|
||||||
|
name: two_view_pipeline
|
||||||
|
extractor:
|
||||||
|
name: gluefactory_nonfree.superpoint
|
||||||
|
max_num_keypoints: 2048
|
||||||
|
force_num_keypoints: True
|
||||||
|
detection_threshold: 0.0
|
||||||
|
nms_radius: 3
|
||||||
|
trainable: False
|
||||||
|
matcher:
|
||||||
|
name: matchers.lightglue
|
||||||
|
filter_threshold: 0.1
|
||||||
|
flash: false
|
||||||
|
checkpointed: true
|
||||||
|
ground_truth:
|
||||||
|
name: matchers.depth_matcher
|
||||||
|
th_positive: 3
|
||||||
|
th_negative: 5
|
||||||
|
th_epi: 5
|
||||||
|
allow_no_extract: True
|
||||||
|
train:
|
||||||
|
seed: 0
|
||||||
|
epochs: 50
|
||||||
|
log_every_iter: 100
|
||||||
|
eval_every_iter: 1000
|
||||||
|
lr: 1e-4
|
||||||
|
lr_schedule:
|
||||||
|
start: 30
|
||||||
|
type: exp
|
||||||
|
on_epoch: true
|
||||||
|
exp_div_10: 10
|
||||||
|
dataset_callback_fn: sample_new_items
|
||||||
|
plot: [5, 'gluefactory.visualization.visualize_batch.make_match_figures']
|
||||||
|
benchmarks:
|
||||||
|
megadepth1500:
|
||||||
|
data:
|
||||||
|
preprocessing:
|
||||||
|
side: long
|
||||||
|
resize: 1600
|
||||||
|
eval:
|
||||||
|
estimator: opencv
|
||||||
|
ransac_th: 0.5
|
||||||
|
hpatches:
|
||||||
|
eval:
|
||||||
|
estimator: opencv
|
||||||
|
ransac_th: 0.5
|
||||||
|
model:
|
||||||
|
extractor:
|
||||||
|
max_num_keypoints: 1024
|
Loading…
Reference in New Issue