Fink is a broker infrastructure enabling a wide range of applications and services to connect to large streams of alerts issued from telescopes all over the world. Fink core is based on the Apache Spark framework, and more specifically it uses the Structured Streaming processing engine. The language chosen for the API is Python, which is widely used in the astronomy community, has a large scientific ecosystem and easily connects with existing tools.
Fink’s goal is twofold: providing a robust infrastructure and state-of-the-art streaming services to LSST scientists, and enabling user-defined science cases in a big data context. Fink decouples resources needed for listening to the stream (online, critical), and resources used for services: scalable, robust, and modular!
We want Fink to be able to filter, aggregate, enrich, consume incoming data streams or otherwise transform into new streams for further consumption or follow-up processing. Following LSST LDM-612, Fink’s ultimate objectives are (no specific order):
- redistributing alert packets
- filtering alerts
- cross-correlating alerts with other static catalogs or alert stream
- classifying events scientifically
- providing user interfaces to the data
- coordinating scientific activity among collaborators
- triggering followup observing
- for users with appropriate data rights, facilitating followup queries and/or user-generated processing within the corresponding Data Access Center
- managing annotation & citation as followup observations are made
- collecting classification and other information gathered by the scientific community
Learning Fink is easy whether you are a developer or a scientist:
- Learn about the broker technology, the science we do, and how to receive alerts.
- Learn how to use the broker or how to contribute following the different tutorials.
- Explore the different components:
- fink-alert-simulator: Simulate alert streams for the Fink broker.
- fink-broker: Astronomy Broker based on Apache Spark.
- fink-science: Define your science modules to add values to Fink alerts.
- fink-filters: Define your filters to create your alert stream in Fink.
- fink-client: Light-weight client to manipulate alerts from Fink.