Vision Transformer (vit_jax)

Use it when

• You have a modest labeled image dataset and need strong classification accuracy by fine-tuning a pretrained checkpoint instead of training a vision model from scratch
• You need to pick among released ViT, MLP-Mixer, or hybrid checkpoints with published accuracy and speed trade-offs before committing GPU budget

What it solves

Not the fit when

• Production inference serving of vision models
• Text or LLM fine-tuning tasks
• Multi-lingual LiT text encoders (not yet released per README)
• experiment-reproducibility
• cost-per-call-too-high

Install

git clone https://github.com/google-research/vision_transformer && cd vision_transformer; Python>=3.10; pip install -r vit_jax/requirements.txt (GPU) or pip install -r vit_jax/requirements-tpu.txt (TPU); install Flaxformer per its repository instructions

Invoke

python -m vit_jax.main --workdir=/tmp/vit-run --config=$(pwd)/vit_jax/configs/vit.py:b16,cifar10 --config.pretrained_dir='gs://vit_models/imagenet21k'; checkpoints downloadable via wget https://storage.googleapis.com/vit_models/imagenet21k/ViT-B_16.npz

Alternatives

No reviewed alternatives recorded yet.