Show HN: Drop-in SQS replacement based on SQLite
500 by memset | 122 comments .
Hi! I wanted to share an open source API-compatible replacement for SQS. It's written in Go, distributes as a single binary, and uses SQLite for underlying storage. I wrote this because I wanted a queue with all the bells and whistles - searching, scheduling into the future, observability, and rate limiting - all the things that many modern task queue systems have. But I didn't want to rewrite my app, which was already using SQS. And I was frustrated that many of the best solutions out there (BullMQ, Oban, Sidekiq) were language-specific. So I made an SQS-compatible replacement. All you have to do is replace the endpoint using AWS' native library in your language of choice. For example, the queue works with Celery - you just change the connection string. From there, you can see all of your messages and their status, which is hard today in the SQS console (and flower doesn't support SQS.) It is written to be pluggable. The queue implementation uses SQLite, but I've been experimenting with RocksDB as a backend and you could even write one that uses Postgres. Similarly, you could implement multiple protocols (AMQP, PubSub, etc) on top of the underlying queue. I started with SQS because it is simple and I use it a lot. It is written to be as easy to deploy as possible - a single go binary. I'm working on adding distributed and autoscale functionality as the next layer. Today I have search, observability (via prometheus), unlimited message sizes, and the ability to schedule messages arbitrarily in the future. In terms of monetization, the goal is to just have a hosted queue system. I believe this can be cheaper than SQS without sacrificing performance. Just as Backblaze and Minio have had success competing in the S3 space, I wanted to take a crack at queues. I'd love your feedback!
New best story on News: Show HN: Drop-in SQS replacement based on SQLite
Show HN: Drop-in SQS replacement based on SQLite
479 by memset | 122 comments on News.
Hi! I wanted to share an open source API-compatible replacement for SQS. It's written in Go, distributes as a single binary, and uses SQLite for underlying storage. I wrote this because I wanted a queue with all the bells and whistles - searching, scheduling into the future, observability, and rate limiting - all the things that many modern task queue systems have. But I didn't want to rewrite my app, which was already using SQS. And I was frustrated that many of the best solutions out there (BullMQ, Oban, Sidekiq) were language-specific. So I made an SQS-compatible replacement. All you have to do is replace the endpoint using AWS' native library in your language of choice. For example, the queue works with Celery - you just change the connection string. From there, you can see all of your messages and their status, which is hard today in the SQS console (and flower doesn't support SQS.) It is written to be pluggable. The queue implementation uses SQLite, but I've been experimenting with RocksDB as a backend and you could even write one that uses Postgres. Similarly, you could implement multiple protocols (AMQP, PubSub, etc) on top of the underlying queue. I started with SQS because it is simple and I use it a lot. It is written to be as easy to deploy as possible - a single go binary. I'm working on adding distributed and autoscale functionality as the next layer. Today I have search, observability (via prometheus), unlimited message sizes, and the ability to schedule messages arbitrarily in the future. In terms of monetization, the goal is to just have a hosted queue system. I believe this can be cheaper than SQS without sacrificing performance. Just as Backblaze and Minio have had success competing in the S3 space, I wanted to take a crack at queues. I'd love your feedback!
479 by memset | 122 comments on News.
Hi! I wanted to share an open source API-compatible replacement for SQS. It's written in Go, distributes as a single binary, and uses SQLite for underlying storage. I wrote this because I wanted a queue with all the bells and whistles - searching, scheduling into the future, observability, and rate limiting - all the things that many modern task queue systems have. But I didn't want to rewrite my app, which was already using SQS. And I was frustrated that many of the best solutions out there (BullMQ, Oban, Sidekiq) were language-specific. So I made an SQS-compatible replacement. All you have to do is replace the endpoint using AWS' native library in your language of choice. For example, the queue works with Celery - you just change the connection string. From there, you can see all of your messages and their status, which is hard today in the SQS console (and flower doesn't support SQS.) It is written to be pluggable. The queue implementation uses SQLite, but I've been experimenting with RocksDB as a backend and you could even write one that uses Postgres. Similarly, you could implement multiple protocols (AMQP, PubSub, etc) on top of the underlying queue. I started with SQS because it is simple and I use it a lot. It is written to be as easy to deploy as possible - a single go binary. I'm working on adding distributed and autoscale functionality as the next layer. Today I have search, observability (via prometheus), unlimited message sizes, and the ability to schedule messages arbitrarily in the future. In terms of monetization, the goal is to just have a hosted queue system. I believe this can be cheaper than SQS without sacrificing performance. Just as Backblaze and Minio have had success competing in the S3 space, I wanted to take a crack at queues. I'd love your feedback!
New best story on Hacker News: Show HN: I made a puzzle game that gently introduces my favorite math mysteries
Show HN: I made a puzzle game that gently introduces my favorite math mysteries
652 by MCSP | 127 comments on
This is the first iteration of a short game I’m making that tries to interactively explain some of my favorite math questions / ideas. My goal is mostly to get the player curious and not necessarily to explain absolutely everything. There were a lot of fun technical parts to building this: - For implementation reasons, it’s much easier if the lines all have integer intersection points with each other. To do this, when a new line is added I “cheat” by rounding intersections to integers and then splitting the old lines at the intersection into new linds (with potentially different slopes) going through the rounded point - I had to draw semi accurate maps of actual places (UK, South America, US west coast) in the HTML canvas using just line segments. I tried a few different solutions, including using SVG data. I ended up using the topojson library to give nice line approximations to GeoJSON maps - I use a simple backtracking algorithm to handle the live coloring of graphs - I use turf.js’s polygonize function to handle finding polygons from line segments (very happy I didn’t have to implement this myself!) - I wanted to make the game as mobile friendly as possible (don’t think I’ve nailed this quite yet) There were also a few tradeoffs I made: - I wanted give links earlier in the game for players to learn more, but I decided to wait until the end to maintain the flow of the game - In order to make the game more mobile-friendly, I generally stuck to maps with a small number of regions (at least for maps people have to interact with them). So for the most part all of the instances in the game are “easy”
652 by MCSP | 127 comments on
This is the first iteration of a short game I’m making that tries to interactively explain some of my favorite math questions / ideas. My goal is mostly to get the player curious and not necessarily to explain absolutely everything. There were a lot of fun technical parts to building this: - For implementation reasons, it’s much easier if the lines all have integer intersection points with each other. To do this, when a new line is added I “cheat” by rounding intersections to integers and then splitting the old lines at the intersection into new linds (with potentially different slopes) going through the rounded point - I had to draw semi accurate maps of actual places (UK, South America, US west coast) in the HTML canvas using just line segments. I tried a few different solutions, including using SVG data. I ended up using the topojson library to give nice line approximations to GeoJSON maps - I use a simple backtracking algorithm to handle the live coloring of graphs - I use turf.js’s polygonize function to handle finding polygons from line segments (very happy I didn’t have to implement this myself!) - I wanted to make the game as mobile friendly as possible (don’t think I’ve nailed this quite yet) There were also a few tradeoffs I made: - I wanted give links earlier in the game for players to learn more, but I decided to wait until the end to maintain the flow of the game - In order to make the game more mobile-friendly, I generally stuck to maps with a small number of regions (at least for maps people have to interact with them). So for the most part all of the instances in the game are “easy”
New best story on News: Show HN: I made a puzzle game that gently introduces my favorite math mysteries
Show HN: I made a puzzle game that gently introduces my favorite math mysteries
645 by MCSP | 125 comments .
This is the first iteration of a short game I’m making that tries to interactively explain some of my favorite math questions / ideas. My goal is mostly to get the player curious and not necessarily to explain absolutely everything. There were a lot of fun technical parts to building this: - For implementation reasons, it’s much easier if the lines all have integer intersection points with each other. To do this, when a new line is added I “cheat” by rounding intersections to integers and then splitting the old lines at the intersection into new linds (with potentially different slopes) going through the rounded point - I had to draw semi accurate maps of actual places (UK, South America, US west coast) in the HTML canvas using just line segments. I tried a few different solutions, including using SVG data. I ended up using the topojson library to give nice line approximations to GeoJSON maps - I use a simple backtracking algorithm to handle the live coloring of graphs - I use turf.js’s polygonize function to handle finding polygons from line segments (very happy I didn’t have to implement this myself!) - I wanted to make the game as mobile friendly as possible (don’t think I’ve nailed this quite yet) There were also a few tradeoffs I made: - I wanted give links earlier in the game for players to learn more, but I decided to wait until the end to maintain the flow of the game - In order to make the game more mobile-friendly, I generally stuck to maps with a small number of regions (at least for maps people have to interact with them). So for the most part all of the instances in the game are “easy”
645 by MCSP | 125 comments .
This is the first iteration of a short game I’m making that tries to interactively explain some of my favorite math questions / ideas. My goal is mostly to get the player curious and not necessarily to explain absolutely everything. There were a lot of fun technical parts to building this: - For implementation reasons, it’s much easier if the lines all have integer intersection points with each other. To do this, when a new line is added I “cheat” by rounding intersections to integers and then splitting the old lines at the intersection into new linds (with potentially different slopes) going through the rounded point - I had to draw semi accurate maps of actual places (UK, South America, US west coast) in the HTML canvas using just line segments. I tried a few different solutions, including using SVG data. I ended up using the topojson library to give nice line approximations to GeoJSON maps - I use a simple backtracking algorithm to handle the live coloring of graphs - I use turf.js’s polygonize function to handle finding polygons from line segments (very happy I didn’t have to implement this myself!) - I wanted to make the game as mobile friendly as possible (don’t think I’ve nailed this quite yet) There were also a few tradeoffs I made: - I wanted give links earlier in the game for players to learn more, but I decided to wait until the end to maintain the flow of the game - In order to make the game more mobile-friendly, I generally stuck to maps with a small number of regions (at least for maps people have to interact with them). So for the most part all of the instances in the game are “easy”
New best story on Hacker News: Show HN: Every mountain, building and tree shadow mapped for any date and time
Show HN: Every mountain, building and tree shadow mapped for any date and time
608 by tppiotrowski | 179 comments on
I've been working on this project for about 4 years. It began as terrain only because world wide elevation data was publicly available. I then added buildings from OpenStreetMap (crowd sourced) and more recently from Overture Maps data. Some computer vision/machine learning advancements [1] in the past few years have made it possible to estimate tree canopy heights using satellite imagery alone making it possible to finally add trees to the map. The data isn't perfect, but it's within +/- 3 meters of so. Good enough to give a general idea for any location on Earth. Happy to answer any questions. [1] https://ift.tt/rY0whLT
608 by tppiotrowski | 179 comments on
I've been working on this project for about 4 years. It began as terrain only because world wide elevation data was publicly available. I then added buildings from OpenStreetMap (crowd sourced) and more recently from Overture Maps data. Some computer vision/machine learning advancements [1] in the past few years have made it possible to estimate tree canopy heights using satellite imagery alone making it possible to finally add trees to the map. The data isn't perfect, but it's within +/- 3 meters of so. Good enough to give a general idea for any location on Earth. Happy to answer any questions. [1] https://ift.tt/rY0whLT
New best story on News: Show HN: Every mountain, building and tree shadow mapped for any date and time
Show HN: Every mountain, building and tree shadow mapped for any date and time
604 by tppiotrowski | 179 comments .
I've been working on this project for about 4 years. It began as terrain only because world wide elevation data was publicly available. I then added buildings from OpenStreetMap (crowd sourced) and more recently from Overture Maps data. Some computer vision/machine learning advancements [1] in the past few years have made it possible to estimate tree canopy heights using satellite imagery alone making it possible to finally add trees to the map. The data isn't perfect, but it's within +/- 3 meters of so. Good enough to give a general idea for any location on Earth. Happy to answer any questions. [1] https://ift.tt/rY0whLT
604 by tppiotrowski | 179 comments .
I've been working on this project for about 4 years. It began as terrain only because world wide elevation data was publicly available. I then added buildings from OpenStreetMap (crowd sourced) and more recently from Overture Maps data. Some computer vision/machine learning advancements [1] in the past few years have made it possible to estimate tree canopy heights using satellite imagery alone making it possible to finally add trees to the map. The data isn't perfect, but it's within +/- 3 meters of so. Good enough to give a general idea for any location on Earth. Happy to answer any questions. [1] https://ift.tt/rY0whLT
Subscribe to:
Posts (Atom)