AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |
Back to Blog
Apache iceberg spark12/6/2023 ![]() The iceberg-nessie module is bundled with Spark and Flink runtimes for all versions from 0.11.0. Getting Started to start a Nessie server. See Project Nessie for more information on Nessie. git-like operations (eg branches, tags, commits).Nessie provides several key features on top of Iceberg: This section describes how to use Iceberg with Nessie. You can find the MinIO UI at where you should see the ‘warehouse’ bucket.Iceberg provides integration with Nessie through the iceberg-nessie module. postgres/data:/var/lib/postgresql/data minio: image: minio/minio container_name: minio environment: - MINIO_ROOT_USER=admin - MINIO_ROOT_PASSWORD=password ports: - 9001: 9001 - 9000: 9000 command: mc: depends_on: - minio image: minio/mc container_name: mc environment: - AWS_ACCESS_KEY_ID=demo - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 entrypoint: > /bin/sh -c "įinally, we can fire up the containers! docker-compose up " postgres: image: postgres:13.4-bullseye container_name: postgres environment: - POSTGRES_USER=admin - POSTGRES_PASSWORD=password - POSTGRES_DB=demo_catalog volumes:. Version: "3" services: spark-iceberg: image: tabulario/spark-iceberg depends_on: - postgres container_name: spark-iceberg environment: - SPARK_HOME=/opt/spark - PYSPARK_PYTON=/usr/bin/python3.9 - PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/spark/bin:/opt/spark/sbin - AWS_ACCESS_KEY_ID=admin - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 volumes:. Must be set but the value doesn’t matter since we’re running locally. Additionally, we’ll need to set the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION environment variables for our MinIO cluster. s3.endpoint= We can append these property changes to our nf in the tabulario/spark-iceberg image by overriding the entrypoint for our spark-icebergĬontainer. The S3FileIO implementation and connect it to our MinIO container. We’ll need to change three properties on the demo catalog to use The file-io for a catalog can be set and configured through Spark properties. If the bucket already exists, the CLI container will fail gracefully. usr/bin/mc policy set public minio/warehouse usr/bin/mc rm -r -force minio/warehouse Until (/usr/bin/mc config host add minio admin password) do echo '.waiting.' & sleep 1 done ![]() mc: depends_on: - minio image: minio/mc container_name: mc environment: - AWS_ACCESS_KEY_ID=demo - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 entrypoint: > /bin/sh -c " Here’s what your docker compose file should look like after following the steps in theĭocker, Spark, and Iceberg: The Fastest Way to Try Iceberg! post. The easiest way to get a MinIO instance is using the official minio/minio image. To learn more about it you can head over to their site If you’re not familiar with what MinIO is, it’s a flexible and performant object store that’s powered by Kubernetes. In that post, we selected the hadoop file-io implementation, mainly because it supported reading/writing to local files (check out this post to learn more about the FileIO interface.) In this blog post, we’ll take one step towards a more typical, modern, cloud-based architecture and switch to using Iceberg’s S3 file-io implementation, backed by a MinIO instance which supports the S3 API. In a previous post, we covered how to use docker for an easy way to get up and running with Iceberg and its feature-rich Spark integration.
0 Comments
Read More
Leave a Reply. |