Spark NLP is a Scala-based NLP library for Apache Spark with thousands of stars and ongoing releases since 2017. It includes many pre-trained models and supports multiple platforms and integrations.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
Spark NLP is a library for Natural Language Processing built on top of Apache Spark. It provides NLP annotations for machine learning pipelines and supports running in distributed environments. It offers pipelines and models for tasks such as tokenization, POS tagging, NER, embeddings, translation, QA, and more, across many languages. It also supports model import from TensorFlow, ONNX, OpenVINO, and Llama.cpp (GGUF).
How it works
The project exposes pre-trained pipelines and models that can be downloaded and used via a Spark-based workflow. It includes a Python start flow using sparknlp.start() to initialize a SparkSession and then download and apply a pretrained pipeline. It supports CPU and GPU usage, and provides documentation for platform compatibility and integration with Spark.
Getting started
$ java -version
# should be Java 8 or 11 (Oracle or OpenJDK)
$ conda create -n sparknlp python=3.7 -y
$ conda activate sparknlp
# spark-nlp by default is based on pyspark 3.x
$ pip install spark-nlp==6.4.2 pyspark==3.3.1
In Python:
from sparknlp.base import *
from sparknlp.annotator import *
from sparknlp.pretrained import PretrainedPipeline
import sparknlp
spark = sparknlp.start()
pipeline = PretrainedPipeline('explain_document_dl', lang='en')
Recent releases
Latest releases include:
- 6.4.2 (2026-06-24): Spark NLP 6.4.2 with BM25 Retrieval, SaT Sentence Detection, and Model Conversion Workflows
- 6.4.1 (2026-05-25): Late Chunking, Multimodal Embeddings, and Smarter Pretrained Workflows
- 6.4.0 (2026-04-07): LLM-Powered Entity Extraction, Expanded Document Readers, and Pipeline Robustness
- 6.3.3 (2026-03-10): ModernBERT Embeddings, Vector DB Integration, and Layout-Aware Document Processing
- 6.3.2 (2026-01-29): Scala 2.13 Support, Layout-Aware Images, and Enhanced LightPipeline Tracking
Traction
The repository has 4151 stars and 743 forks. It shows 23 open issues.
Behind the repo
The project is led by John Snow Labs and includes a focus on NLP and machine learning pipelines with Spark integration.
Caveats
License: Apache-2.0. Created 2017-09-24. Last push 2026-08-04. Language: Scala. Topics include bert, entity-extraction, language-detection, lemmatizer, llamacpp, llm, machine-translation, named-entity-recognition, natural-language-processing, nlp, onnx, part-of-speech-tagger, pyspark, question-answering, sentiment-analysis, spark, spell-checker, tensorflow, text-classification, transformers. Open issues: 23.