The elephant in the room is that AWS Babelfish could be a viable alternative to PostgreSQL. Babelfish is currently a fork of PostgreSQL that is compatible with Microsoft SQL Server’s TDS wire protocol and T-SQL language. The article proposes adding hooks for wire protocol extensions eliminating the need for separate forks. Adding new languages is already supported (EDIT: incorrect, see mattashii's comment).
One of my fears. That unless we, as the PostgreSQL Community, do something about it, and embrace the efforts that AWS started doing and I hope will continue, it would effectively end up being a fork. And a fork with a collaboration on GitHub, which many would prefer to contribute to rather than current's Postgres contribution mechanism.
And if that's the case, I wonder if it could become the MariaDB, or the LibreOffice, of Postgres. I hope we will be able to do the best thing and have the contribution that Postgres is receiving integrated and merged into Postgres.
Disclosure: I work for Amazon where I build AWS infrastructure, but these opinions are mine alone.
Personally, I think that the word "fork" has a lot of historical meaning in the world of Free and Open Source Software. To me, "fork" means that there is a "vote of no confidence" in the direction of the people who are making decisions about open source software. That clearly is not the case here. The PostgreSQL core team has a very good reputation of being collaborative, and thoughtful, in their work to make PostgreSQL the best database it can be for the community at large, and not build it to benefit any particular commercial interest.
Often software development happens on branches. These are experiments, not forks. Over time, good ideas are propagated, and grafted, between the branches. These are not "forks" to me.
I see this as a very open and collaborative approach. To announce such a big contribution to Postgres, and be it open sourced when published, is very welcome, and I believe one of the boosts, as I mentioned in the post, that Postgres needs.
I'm also convinced that Babelfish and PostgreSQL will be properly integrated, and this will be a great win. I consider it too a development branch more than a fork, which certainly has "no confidence" connotations. But there's still some worry that it would become one, one day, if there's no sufficient effort towards its integration --as the conversation that I wanted to spark here was almost non-existent up until recently.
This actually brings the topic of more explicitly branch development in Postgres, but it's a different story ;)
> To announce such a big contribution to Postgres, and be it open sourced when published, is very welcome
But the source code hasn't been published. As I said to you today on pgsq-hackers, that's why it hasn't been taken seriously - that's just how it works.
> But there's still some worry that it would become one, one day
Who are you attributing that concern to, exactly?
You're basically claiming that it will be a great opportunity. But you're also claiming that it might be an existential crisis. Do I have that right?
Have you considered the possibility that it might actually be neither of those two things?
And maybe that's just how it shouldn't. Technology moves fast, very fast. Markets do so. Users, you get it. Competing technologies, undeniably.
To delay a discussion for months just because the source is not published is in my opinion only reflecting on part of the opportunity/problem. I'm calling for an strategic discussion, which is higher level than a code-only, limited, partial analysis.
> Have you considered the possibility that it might actually be neither of those two things?
Sure thing, like anything in life, it may be neither. But what I know for sure is that a) we as the PostgreSQL Community need to mitigate risks and plan for them; b) leverage (or transform into!) opportunities as soon as we identify them.
There is practically no information about Babelfish available to analyze. You can currently request a preview of Babelfish if you have an AWS account, though I'm not aware of anyone having even that level of access. There is a single page placeholder on a Github.io that says "Babelfish for PostgreSQL will be available on Github in 2021".
And so I genuinely don't know what a strategic discussion really means here. How much can be decided about something based on a vague press release?
It is interesting to think about this in the abstract, but I think that Peter is right. The real collaboration only comes with getting the code published, design proposals on extension points in the core engine, and so on.
This isn't the sort of thing that is decided with blog posts, but personally I am glad to see there is interest in this project.
There is information about Babelfish, including reInvent keynote, website and other sources. Here you have a 30min session about Babelfish for PostgreSQL: https://www.youtube.com/watch?v=DcGMoxPCNcQ
I'd also love to have more, architectural/design documents, the code itself, etc, and I'm sure they will be sooner than later released.
But that doesn't prevent us, as I said, to discuss the items I suggested above. They do not depend on the source code, at all.
And independently of this, there is already one thread on the -hackers mailing list that provides some information and proposes to introduce a technical improvement that not only would ease the Babelfish integration, but will also provide significant benefits for PostgreSQL itself, allowing for other protocols to come (including a potential v4 and an HTTP-based protocol): https://www.postgresql.org/message-id/CAGBW59d5SjLyJLt-jwNv%...
I think Github has softened the term "fork" a lot, with putting all personal branches into forks.
To indicate a non-collaborative fork, I usually use the term "hard-fork", which I know from the blockchain world, where it has a similar "vote of no confidence"/"we are starting our own project with new leadership" meaning.
I would add that depending on context, the hard part can be implied, particularly of the name is changed. E.g. "LibreOffice is a fork of OpenOffice." It is very context dependent, whether I interpret "fork" to mean hard or soft.
AWS has enough scale to have its own gravity and will likely run development forward regardless of .org, it would be GREAT if .org could actively work to integrate where possible the open source options here.
I remember some early efforts of google to contribute wakelocks and other features to the linux kernel - so annoying when that was shot down and created a pretty significant split, and the commercial entity wasn't going to hold off on wakelocks because a subsystem maintainer didn't like it - so android starting going off into its own land.
Wakelocks also ended up proving a reasonable power management approach (despite complaints they didn't use existing linux mechanisms etc).
There was a big push ultimately to get wakelocks into mainline and drive some convergence.
This indeed is the concern. I don't think most people are going to care if it is PostgreSQL.Org PostgreSQL, or Babelfish for PostgreSQL in the long run. What they will care about is modernity of the development process and who has the "cool" features.
When you look at something that has nothing to do with the Babelfish functionality but would absolutely help PostgreSQL, consider PHOT. PHOT (Partial HOT) is a great opportunity for PostgreSQL that Amazon is trying to give back. However, I wouldn't be surprised if the .Org community were to not accept the patch, the feature would end up in Babelfish. If that happens, we now have a viable fork which has 100% compatibility++.
This is true only to a degree: Adding new procedural languages is supported, but support for new query dialects (i.e. the main contact surface of most applications with an SQL RDBMS) is not currently a supported feature.
As mentioned in the psql-hackers thread from the article, currently the postgresql-specific parsed syntax tree is passed through many components, and to replace the syntax you'd have to alter and recompile the core application.
The opening paragraph of the article says " Amazon AWS announced Babelfish" and links to https://aws.amazon.com/blogs/opensource/want-more-postgresql... from the AWS blog. This all seems accurate to me. Is there some other reason why we should take AWS out of the title? Branding is not a reason that HN readers would support, and I think if we took AWS out for that reason, the community would accuse us of trying to obscure the origins of the project. Internet people are trigger-happy about this kind of thing, and too quick to accuse us of serving $BigCo agenda as it is.
I think we are trying to establish the brand of this project in these early days, and to have that brand distinct from AWS. I see this like "Mozilla" was separate from "Netscape" in their early days [1] (note, this link will not work when the referrer is from HN. Archive link provided for convenience [2]). It is a project that AWS is sponsoring and stewarding, but from what I currently know, I think the intent is for it to be community-centric in the long term.
If successful, Babelfish will grow a community, and work harmoniously as an extension of PostgreSQL. And including "AWS" in the brand of the project may hamper the efforts to build a community around it (in addition to complications of using a brand / trademark like "AWS" in an open-source project).
The TL;DR is: "AWS" is not part of the branding of Babelfish for PostgreSQL, and that is intentional. The reasons for this aren't about obscuring the origins or initial sponsorship of the work, but are rather about establishing a separate identity.
AWS is currently keeping Babelfish closed (Apache 2.0-licenced as it may be, no code has been published, making it little better than ordinary proprietary software), so I see no reason to not add the 'AWS' to its name as of this moment, as thats the only place you can get and use Babelfish. Whenever that situation changes, my standpoint will likely change as well.
I worked with SQL Server, Oracle, MySQL and many other DBMSes. Nothing can beat PostgreSQL.
Oracle's SQL Developer is the worst IDE for databases I ever seen. Oracle DB doesn't support "TOP N" or "LIMIT N" notations and many other useful features...
I like SQL Server more than Oracle and Microsoft provides 2 good IDEs for it (SQL Management Studio and Azure Data Studio). But it has some bad limitations - you can't define ON ... CASCADE on more than one path. You also can't create CTEs with inserts/updates.
PostgreSQL doesn't have a good IDE out of the box. I heard that Azure Data Studio now support it, but didn't try it yet, because I already paid for a commercial IDE that supports PostgreSQL. PostgreSQL also doesn't have a good high availability tooling, but all major cloud providers do this for you. Considering that you get it for free, it is a much better choice than Oracle or SQL Server.
I really love using dbeaver[1] as a Postgres IDE. It requires a small bit of configuration for me to use it (why are the defaults to automatically lowercase all of my SQL?!). Other than that it's excellent!
It's a surprising default - for better or worse history made the convention that SQL KEYWORDS be uppercase (when pretty printed). I assume it stems from case being irrelevant and a time of green screens and monochrome printouts - but it's still the most common formatting/presentation of sql?
Working with the big 3 on a regular basis, DataGrip is the best tool I've found. (It lacks some management features, but it's been years since I needed those)
I'm working with many SQL Server, MySQL, PostgreSQL, SQLite and soon Mongo versions daily, and I have to say Datagrip is worth it's price tag easily. It is the only non-open tool I work with because it really makes work with multiple databases a lot easier.
> Oracle DB doesn't support "TOP N" or "LIMIT N" notations and many other useful features...
Instead of those nonstandard notations Oracle supports SQL-standard notation OFFSET and FETCH clauses (Postgres supports both LIMIT/OFFSET and OFFSET/FETCH, but not TOP.) OFFSET/FETCH is, IIRC, slightly more powerful than the nonstandard ones.
Devart makes good SQL IDEs. They came out with one for Postgres ... recently? In the last year or two. If it's similar quality to their mysql ide, I'd recommend it.
In my opinion, this is the inevitable conclusion of Amazon's sudden increase in direct involvement in OSS projects. With cases like elastic search close in the rearview mirror (I know that there were reasons for this, but it was a start), It's hard not to come to this conclusion.
Once the Amazon tentacles have reached down into every major staple of the OSS economy, it's difficult to predict the end result, but one thing is for certain, it will benefit Amazon more than it benefits anyone else, lest Amazon breaks its promise to its shareholders.
Amazon has demonstrated a pattern of controversial vertical integration, by using internal analytics to determine which of their own merchants' products they should undermine. To believe they would not do the same thing in their largest and most important division would require tragic ignorance.
Disclosure: I work for Amazon, where I build AWS infrastructure. But these are my own opinions and experiences at work, and I am not speaking for my employer.
"Embrace, extend, extinguish" has never been uttered in any conversation I've been part of at Amazon. It simply is not part of the philosophy, or culture. We obsess about customers, and customers do not like the things that they depend on, like open source software that they've built their business on, to be damaged.
Rather, they want to do more with open source software all the time. And many want to find ways to get off of software that is constraining them, or causing unplanned stress or surprises when their costs suddenly change due to arbitrary license changes.
The promise here is to customers, and customers tell us that community-led open source software development is better for them. And this is no surprise. As a software developer, and a long time FOSS advocate, I believe it too.
By listening attentively to customers, I believe we will continue to build a business that we, and our shareholders, will be proud to be a part of, and will provide long term growth.
With no disrespect to you, this comes across like a pre-canned response from the Amazon Anti-Anti-Trust legal team...
I mean, your second and third paragraph could be the introduction to an internal presentation about "why we at Amazon should embrace, extend, and extinguish OSS". The things you listed are precisely why Amazon would want to destroy OSS, Because OSS is a threat to anyone offering proprietary software, and anyone who has built their empire on OSS will inevitably need to shift to proprietary software to retain relevance in a world where OSS software "eats the world" by, for instance, commoditizing the infrastructure layer...
In other words, just in time for AWS to become less relevant due to the enormous gains in open source infrastructure as a service tools, Amazon will swoop in and help to destroy the most useful parts of the open source economy.
It's easy to look at this with a naïve optimism and think "but these are the exact kinds of things that helped AWS grow to begin with", To which the obvious reply is "yes, at the bottom end, but Amazon has been focused on enterprise more and more over the past five years, and licensing agreements with other big tech companies have changed the offering and the prospects of future offerings". All you have to do is look at YouTube to see another example of this. YouTube built its user base on small contributors, but as soon as it reached a plateau it began to focus on the old guard, main stream media, Hollywood, etc., unfortunately for the little guys. This is the same pattern every company takes once they get big enough to play competitively with the big boys.
If you understand why I have this viewpoint, then have I convinced you to have the same viewpoint? Or, have I given away some thing about my internal reasoning mechanics to suggest that I would have erroneously come to my conclusion, perhaps based largely on blind cynicism?
Personally, I try to keep a "curious" outlook, and to regularly try to disprove my beliefs. To me, one way to do this is to listening closely to criticism from viewpoints and perspectives that are different than mine.
Based on my own experiences, including engaging in very helpful dialog with critics so far, I am not convinced that the same kind of anti-FOSS campaigns that we saw, with ample public evidence, from actors like Microsoft and SCO of the past, would be in the interest of Amazon or its customers. It would be the opposite.
But I think it is right, and it should be expected, to continue to scrutinize Amazon, like any other large organization. I try to judge based on actions and evidence, rather than what-if speculation.
But besides that, I don't think Microsoft engineers in their day to day work ever mention embrace, extend, extinguish. Amazon's motive is profit. They're darn good at it too as we've seen. But that's all it is. If in the process they destroy an OSS company, as they've done, they don't care, because now they have their source and no competition.
> "Embrace, extend, extinguish" has never been uttered in any conversation I've been part of at Amazon.
We are not talking about internal corporate PR. Enron also had great set of values [1] that were preached to the employees. It's about the actions, not the words uttered.
I am not reciting corporate PR. As I said, I'm not speaking for AWS. I only speak for myself, and I'm not saying anything that I don't believe.
You'll find elsewhere in the comments [1] that on actions, not words, I agree with you. "I try to judge based on actions and evidence, rather than what-if speculation."
I've been working for AWS across different teams for some time now and can't confirm that. There are a LOT of folks with +5 years tenure. In the US, the average tenure of workers aged 25-34 is 2.8 years, so you'll obviously see some turnover and some teams might see more than others - but the attrition rate is not really high across the company. In my opinion, we're hiring mainly because we are still growing really fast!
You might be on a solid team, but AWS is widely regarded as being a complete meat grinder. We've hired so many devs from AWS in the past 18 months (even pre covid) and they all have the same story. 12-14 hour days at a minimum, toxic leadership and managers, incredible pressure, tons of tech debt and mostly doing ops work.
I'd say the trend for AWS is leaning "do not join". Anecdotal? Of course, but with enough stories it becomes a statistic and not an anomaly.
Hm. I will obviously appear to be biased, but isn't the collection of anecdotes that you've gathered a manifestation of some kind of inverse survivorship bias? Within the total population of people who work or have worked at AWS, the largest group of people with a negative experience will be in the subset of the population that has left the company? I really believe that companies the size of Amazon will inevitably have a certain percentage of people who had negative experiences - just like the company has negative experiences with individuals.
Like I said in my other comment in this thread, I'm sure that there are some teams that don't have a good culture but it also comes down to the individual in many cases. I guess it comes down to individual preferences, some folks really don't want to work at Google, others dislike Facebook, some don't like AWS, etc.. But at the end of the day, all of these companies still attract a ton of incredibly talented people and I personally don't see that trend ending soon.
Few companies have entire websites dedicated to how poor their experience was at said company. Amazon is the only company I've ever heard of in tech where people have devoted their spare time and money building resources to warn others about how bad and toxic their experience was. Microsoft doesn't have this. Facebook or Google don't have this. It's actually mind blowing when you think about it.
Imo, Amazon is a special level of toxic and abusive and it shows by their reputation in the industry and how high their turnover is. That's what I mean by anomaly vs statistic. When thousands (tens of thousands?) of your former employees have horror stories, it starts to build a strong narrative than a few bad teams or a bad manager. Or you start to question if the ratio of bad teams is that high, what is wrong the the company as a whole?
Does it mean your experience at Amazon is wrong or incorrect? Absolutely not. But your experience isn't the norm.
Out of curiosity, I just did a few searches along the lines of "is MSFT/FB/GOOG bad place to work" and for each of them there were several results/websites about how each of these companies is the worst place to work. If you look at the largest publicly available data set, Glassdoor, AWS does quite well - better than Microsoft and as well as Google. I have not worked for Amazon outside of AWS, so I can't really comment on that. However, as Amazon has more than 1,000,000 employees (400,000 of which were hired in 2020), I'd argue that you get more negative stories simply because of the larger population? Comparing Amazon to MSFT (163,000), FB (50,000) or GOOG (135,000) without taking its ginormous scale into account seems to be comparing apples to oranges. In terms of scale and typology, it would probably be better to compare AWS to these companies and I believe that the publicly available data paints a different picture than what you describe.
I'm not trying to argue with you here, I've just had a quite few discussions in recent times with people who had very strong negative opinions about MSFT being a place for the lazy, Google being a place for the special snowflakes, Facebook being a place for the cultish, AWS being a place for those looking for a stepping stone in their career, etc....and I think that all of these companies actually are great places to work but there will always be a few percent who absolutely hated their time there.
Just filled with stories of people that felt extremely wronged. Are there stories of people that with terrible experiences at other companies? Of course. Are there people that have had good experiences at Amazon? Absolutely.
Does anything like the above exist for any other company? Not that I can find.
That's crazy and really speaks to the difference. If you believe that Amazon is "just another tech company" in how they treat their employees, I'd probably say you're turning a bit of a blind eye.
I can't figure out why you're putting so much stock into that site. It's a random Google Sites page that looks like it took about an hour to set up. In the past 1.5 months, it's gotten a total of 5 submissions. 5 submissions from a company of over a million employees. And of those 5 submissions, 2-3 are from people who do not even work at the company. One of the submissions literally starts with "I've never worked at the company", and one of the others is a NIMBY upset that Amazon is building a warehouse in their town.
If you're telling me that in 1.5 months, a company of 1+ million people only generated two complaints from employees, then that does not at all support the point you are making that Amazon is terrible to work for. That is an astonishingly low complaint rate.
>Does anything like the above exist for any other company? Not that I can find.
Have you never heard of glassdoor? Blind? There are plenty of sites that have literally hundreds of times more complaints against tech companies than what is shown in the site you linked.
Blind completely corroborates that site. If you search "Amazon pip" or just "amazon" you'll find huge numbers of posts with terrible stories. More so than other tech companies. If anything the evidence looks even stronger if you consider Glassdoor and Blind. For example, Amazon's rating is 2/5 stars, the lowest of the large tech companies by far.
Yea, if you search for a term that is related to people having poor experiences at Amazon, on a website that is primarily only used by people who are unhappy with their jobs, of course you're going to find posts from people having poor experiences at Amazon. Duh. If I search "microsoft sucks" on blind I get a bunch of posts about how working at Microsoft sucks. If I do a Google search for "working at Google sucks", what are all of the top results? It's a bunch of articles about how working at Google sucks. If you search for bad reviews about a company, you are going to find bad reviews about a company. That's how search works. It's confirmation bias.
Most of these stories are about Amazon and again, at the scale that Amazon operates I think that this would be expected? With 1,000,000 employees, if you assume a yearly turnover of just 1% that's 10,000 people leaving the company and I'm sure there are quite a few people in that group who had a bad experience - particularly as this includes the retail operation, which definitely had a bad reputation (most stories on the website that you linked are about retail teams). Post the NYT story in 2015 a LOT of things changed in retail, but I'm sure there are still issues.
Not trying to be pedantic here, but this thread was about AWS and not Amazon, so I will just reiterate that I can't personally comment on Amazon as a whole, I can only comment on AWS and I think most people would agree that AWS can be put into the same category as FB, MSFT, GOOG, etc. I have highlighted this distinction several times, but you seem to not want to engage in a conversation about AWS and rather discuss Amazon as a whole. That's fine, but as AWS and Amazon are really two different companies, you are then comparing apples to oranges and in that case you should compare Amazon to e.g. WalMart?
Mm, but you're in management. All the Amazonians I've talked to have indicated that it's a very rough environment and they were glad to get out. Maybe you get a distorted view due to the power imbalance between you and your peers?
I'm an individual contributor in product management, so not in people management. Amazon / AWS is a challenging work environment, just like most other tech companies. In my opinion, that is also why so many people want to work for FAANG & Co, as you generally get to work on really interesting but also challenging things. I'd say that bursts of intense working periods are par for the course in FAANG & Co, but it also depends on your own ability to handle our self-service culture (which I find empowering). The culture is certainly peculiar but each large tech company has their own culture that works really well for some and doesn't jive with others. Just speaking for myself here, but I have learned a ton professionally and personally, work(ed) with inspiring people and had the best work-life balance of my career so far. Don't want to sound like a shill here - I'm certainly aware that in a company the size of Amazon there will be good and bad pockets, I just happened to work with good teams so far.
You unfortunately not only sound like a shill, but like a shill who has read Amazon's literature. I have a few questions that are worth reflecting on; I also asked them of myself when I worked at a FAANG:
* How many hours/week do you spend at your work machine?
* How often is a sprint crunched; how often does the sprint manager exhort sprinters to complete the board?
* How many incidents/quarter does your team have? How often does an incident route through you?
* How many hours/quarter are you oncall/onduty?
* How often do you have anxiety on Sunday night about going to work on Monday?
The answers should be at most 40, rarely, 12, 60, and rarely. Ideally, they would be 20, never, 3, 120, and never. Note that we don't require less oncall; we require less stressful oncall.
> The trends for the last 8 years are clear: Oracle and SQL Server are constantly declining in popularity; MySQL is slightly declining; and PostgreSQL is clearly growing in popularity. But while PostgreSQL almost tripled in popularity in these eight years, it is still far behind the other three.
It's great to see Postgres growing, but I often wonder why it isn't growing faster, especially when compared to MySQL. Without getting into a religious flame war, I'm curious what the HN thinks about that?
People don't need those white-label commodity cPanel or Plesk shared-hosting or VPS services to run SaaS applications anymore - it's far better for everyone[1] when they switch to a major cloud provider, namely AWS, Azure, GCP, etc.
I argue that most web/saas devs today grew up with - and tinkered-with - said white-label services: which means they got their experience with MySQL because it was part of the stock default configuration for all those web-host accounts: with PHP, maybe Perl, and many preinstalled web-applications like phpBB, Wordpress, Coppermine, phpNuke, and so on... and that's probably a solid 15+ years of commonplace web-hosting market saturation (thinking 2001 through 2016, with 2014 being the tipping-point for AWS/Az/GC being the home of SaaS).
You'd be in high-school age-range (14-18?) and get bored of simply running other peoples' programs on your web-space, and you saw these web-applications were written in PHP so you'd follow some series of online tutorials for PHP and when it inevitably leads to databases they assume you have MySQL - and MySQL (at the time) had not only the widest installbase, but also had a far more forgiving SQL engine than anyone else. While PostgreSQL was often only a few clicks away via quick installers built-in to cPanel/Plesk/etc, MySQL was a safer-bet by everyone involved.
...but those days are past: as a personal anecdote: at no-point in the past 10 years have I come across any "serious" PHP web-application software intended for private or on-prem usage since vBulletin or SVN.
> People don't need those white-label commodity cPanel or Plesk shared-hosting or VPS services to run SaaS applications anymore - it's far better for everyone[1] when they switch to a major cloud provider, namely AWS, Azure, GCP, etc.
I think around that time, postgres was handicapped because of performance reasons. It was considered slow because of ACID compliance and MySQL was all the rage with MyISAM.
If only postgres won the marketing battle back then...
PG also shipped with EXTREMELY conservative defaults for buffers/etc., so it ran like total ass 'out of the box' for years. A quick change to buffer allocations and things in the config file made it fly, but you had to 'know' that was a thing, where mysql just...didn't need that tweaking.
> at no-point in the past 10 years have I come across any "serious" PHP web-application software intended for private or on-prem usage since vBulletin or SVN.
There's Phabricator [0], which incidentally uses MySQL.
Enterprise dba shops have deep expertise with oracle (and sometimes sql server). I have personally run into serious resistance from these teams when I explicitly ask for postgres. They try to warn me that oracle is the recommended option for real production use cases because of the deep in-house expertise and that I am pretty much on my own if I go with postgres. Most people back off at that point.
I've seen this so much, and can empathize with them. When you have a product that took you years to master, the value that you bring to your employer is wrapped up in that product. So it isn't so much that someone is a DBA, they are an Oracle Database software specialist. You've learned all the quirks so that when something goes south, you "have the answer". So naturally you will have a number of reasons why another product isn't as good.
I feel the same way with Unix/Linux. At one point I was told that my organization may be migrating away from Linux, and redeploy everything under Windows. The management layer really couldn't understand that they would lose nearly every Linux admin on staff (that happened to also have deep understanding of their apps and infrastructure). "Why would someone have a problem with switching platforms -- we'll provide you the training" was their answer. Fortunately we were able to come up with a number of technical (and cost related) issues that the project was shot down.
At sufficiently large companies, deep in-house experience with a given DB technology may be roughly equivalent to a startup engineers deep experience on their companies current monolith.
Companies with large and complex db installations often need the complexity for various reasons that are difficult to escape, even if you're working on a green field project.
PG over mysql is obvious IMO. PG over mssql, not so obvious. People often forget the 'management system' part of RDBMS. MSSQL provides an enormous amount of tooling in the typical enterprise install. Analysis services, data tools, etc... provide a lot of value beyond a data store.
Of course if you don't need any of these additional tools or can't afford them, then yeah pg is a great choice.
Cost is a really solid argument against MSSQL. Even at Azure, MSSQL costs ~6x as much as the open-source databases. For even relatively small workloads, it's not hard for the MSSQL license to cost as much as an FTE.
Aside from the raw cost, the licensing model and cost considerations can drive design decisions. In my experience, most shops that use MSSQL do so because nobody on staff has experience with other databases.
Sure, but we're talking about enterprises. Back when I could walk down the hall to the datacenter, paying $30k for the hardware and $60k for licenses was normal. Every time I bought a new server I tried to make a case for pg or mysql, but the cost to replace SSIS, SSAS, or SSRS either with another product or to build a replacement was always more.
That doesn't match the economics at any company I've seen. Dev payroll of $1mm/yr gets you something like 6 devs or fewer in the US.
I have yet to encounter a team that would rather have SQL Server than another developer given the explicit choice between the two (assuming easy migration, etc.).
I've only interacted with Analysis Services and Reporting Services for maybe five minutes, but the impression I was left with is that they're extremely clunky, even by Windows standards. I also remember a weird situation with SSL wildcard certificate configuration, where the interface said the configuration wasn't applied, but it actually was.
My client subcontracts this to a supposedly "expert DBA" shop, and they always seem to take ages to do what look like simple things on the surface. (I rarely if ever interact with them, so I don't know whether they're actually competent or not.)
Tools like SSIS, SSAS, SSRS were hard to replace for less money for a very long time. There are products out there now, but they are not cheap either. I still think SSIS was one of the easiest and most flexible to use ETL tools out, and it was included with MSSQLs enterprise license.
> Analysis services, data tools, etc... provide a lot of value beyond a data store
They probably do if you have certain needs, but at lots of places they seem to be the basis for solutions that would be far less complicated (and maintainable with a far more common skillset) without them.
Sorry, OLAP and ETL are real problems that need to be solved, SSAS and SSIS were (and still are in many cases) as good as the next tool for solving them, especially with their enterprise features like security and auditing/logging, which are almost always nonexistent in opensource / homegrown solutions.
Tools like Airflow and Druid are just now getting to where those tools were in 2005 ... unfortunately those tools haven't done much since about 2015 when Azure became the new hotness.
> Enterprise dba shops have deep expertise with oracle (and sometimes sql server).
IME, that “deep experience” often means “a very expensive support contract and internal technical staff who walk through decade+ out of date cargo cult rituals by rote without understanding the rationale and why it has been gone longer than they've been employed”.
Or the same issue form the other side. It's damn near impossible to hire a good DBA in some areas and so if you find someone who has deep magics with MySQL then you go with MySQL. In a huge complex app they're all good options except for Mongo and so going with what your team knows is far more productive.
A significant annoyance is that permissions cannot be encapsulated within a database but are global to the instance, requiring extra consideration at every step, from the very logic of your migrations, to how you dump/restore.
Try the lock down e.g. a user to only allow reads. Not easy, when testing if it‘s working you will see so many „strange“ behaviours e.g. you can still create tables in public namespace, databases created before/after (dont remember which one) your user are accessible etc...
Postgresql is following a blacklist access model which is really hard to get right, a whitelist approach like mysql would be much easier.
> Postgresql is following a blacklist access model which is really hard to get right, a whitelist approach like mysql would be much easier.
I don't think this is quite right. When a new user is created in Postgres, I don't believe any permissions are implicitly GRANTed by default aside from CONNECT. And I believe the only default privileges granted on a given database are to its owner. Is it possible your experience has only been with configurations in which your PG user was either the owner of the database or a superuser?
My statement was not correct, you are right. But every user/role in postgres has by default the rights to create objects in the public schema and to connect to every database [1]. The only way to remove this is to remove the rights on the public role affecting every other user. In this case the mysql concept is better as every new user has by default absolutely no rights.
In particular, note the justification for the default:
> Keep the default. All users access the public schema implicitly. This simulates the situation where schemas are not available at all, giving a smooth transition from the non-schema-aware world. However, this is never a secure pattern. It is acceptable only when the database has a single user or a few mutually-trusting users.
I agree that the whole PUBLIC thing is pretty confusing, and I doubt anybody would design it the way it is if we were start from scratch.
Note that you can change the set of default privileges, including the grant to PUBLIC. E.g.
ALTER DEFAULT PRIVILEGES REVOKE ALL ON TABLES FROM PUBLIC;
> The only way to remove this is to remove the rights on the public role affecting every other user. In this case the mysql concept is better as every new user has by default absolutely no rights.
To the point number two: even though I love to use PostgreSQL, when I hit the point where I need pgBouncer and its transaction mode, things get quite ugly especially if you've been using a lot of prepared statements. Now you need to clean the statements from the connection after every use or get conflict errors if another connection using the same connection from the bouncer wants to declare a statement with the same name.
Or you just use the text protocol and escape your parameters in the app...
Personal guess: because switching DBs is expensive and if it works, why bother?
I personally LOVE pg over all other choices, but I've got some 5-year-old apps sitting on a little mysql DB and no intention to move them. Dbs are small, performance is good enough, cost of instances wouldn't be very different on PG, but cost of moving everything over is probably a week or two of work that would be pretty hard to justify given the current situation.
1. MySQL is offered as the default "open source" database in most discussions. This puts the onus on Postgres to people who go "I need a better database", rather than people going "I need a database" in a "Nobody got fired for choosing MySQL" way.
2. Companies looking for enterprise support don't see anyone as high profile as Microsoft, Oracle, and err, Oracle supporting Postgres.
(Whether ongoing interaction with Oracle is a net positive is of course debatable. At a previous company our only interaction with Oracle was them being inflexible and expensive that ultimate meant we had a Oracle DB on prem serving a AWS hosted service because it would have cost too much to put it in AWS. Note, this was many years ago, before AWS had a compatible offering, and apparently Oracle even have loosened up a bit here since.
MySQL/MariaDB have several relatively easy ways of building multi-master setups. Postgresql doesn’t really have a that great solutions for master-master replication. Our customers go for MariaDB to get master-master replication
We see some customers building new solutions on Postgresql, but for large setups, it’s still MariaDB/Galera or Oracle and there isn’t a big push to drop Oracle.
Would downvote if I could, PostgreSQL has Patroni, for example, the newest replication management tool, which works. They got rid of the requirement for an external DCS with the latest major release, so.. it's easy to set up and manage
When PostgreSQL would get a built-in replication tool would increase it's popularity a lot, but I guess the core team wants to focus on the DB, and management tools are left to the others to develop.
IMO replication is a core feature of a DBMS (like transactions or SQL support), the fact that you need to install obscure third-party tools to bring Postgres into the modern age is bonkers.
At my current gig, we've been burned pretty badly by scalability problems with MySQL, and the prevailing wisdom is that new functionality and services that have to deal with data at scale should be built on Cassandra. My cries of "we wouldn't have this problem with Postgres" have mostly fallen on death ears.
Historically, MySQL has been better at single concurrency setups, like your average Wordpress site whose main reader is the person posting to it. PostgreSQL is usually much better in high volume scenarios.
For instance, suppose you have a query that requires a full table scan. In PostgreSQL, that query comes in and the DB starts the scan. Now, a second such query comes in. PostgreSQL notes “I’m on row 10,000” and continues running one single scan, sending results to both queries. When it reaches the end of the table, it marks the first query as complete, then goes back to the beginning of the table and starts scanning again, sending results to the second query up until it reaches row 10,000. Now the second query has seen every row in the table and it’s finished.
Now imagine 100 such queries arrive. Rather than doing 100 full table scans, PostgreSQL satisfies all of them concurrently, taking at most 2 full scans (assuming the 100th queries joins in on the very last row of the first scan).
PostgreSQL has a million such optimizations under its hood that make it happily chug away even when it’s getting slammed. You can vertically scale a single-instance PostgreSQL server a lot higher than many people would believe.
> PostgreSQL has a million such optimizations under its hood that make it happily chug away even when it’s getting slammed
So does MySQL. Facebook, Google, and Percona have contributed many upstream patches to harden and improve MySQL over the years. MySQL also has much better observability to find issues causing problems, from index_statistics to hunting down individual queries causing locks.
I agree that Postgres is by far the better engineered product, but it's not necessarily the better RDBMS product because of that. MySQL is battle tested in a way that Postgres isn't (yet).
In my cases I found that mysql was faster for “select * from <table> where <unique key>=<blah>” — but for anything more complicated than that, postgres won (admittedly I haven’t checked recently - it was around 10 years ago that my 500-concurrent-user site was melting mysql, so I switched to postgres, which is now happy with 15,000 concurrent users - I’m mostly just responding to “historically more performant”)
MyISAM was historically notably faster for simple SELECTs on the primary key and SELECT COUNT(*).
InnoDB had ranged from slightly faster to much slower than Postgres depending on exactly what you’re doing. Once you’re doing more than trivial queries the MySQL optimizer tends to fall apart, often in weird ways with arbitrary thresholds where performance goes from decent to terrible after a table grows by 5%.
We have a few large (not large in the scheme of things, but large for MySQL) tables that we can't add columns to without locking the table for an extended period. In Postgres this operation is instant no matter the size of the table.
This not the only problem, but it is the most pernicious
Outside the context of db comparisons, and in relation to the specific case, if you don't have triggers/foreign keys on a given MySQL table, Gh-ost¹ solves the DDL locking issues.
Are you sure? Based on the official blog post¹, the INSTANT algorithm is new to MySQL 8.0. The manual of MySQL 5.6 indeed doesn't show any "Instant" option in the list of column operations². AFAIK, previous to MySQL 5.6, the fastest algorithm was INPLACE, which copies the data.
Adding a null column on PostgreSQL is instant. Import your new column data then apply defaults and constraints. This is how people have avoided length locking periods on production databases for years.
I worked at a place that was running SQL Server 2012 in compatibility mode with an older version. The place had a pretty high turn over rate, so a lot of the expertise is in the existing tech rather than the people. It was hard enough to try convince them to upgrade SQL Server (I wanted to leverage row level security to get away from having a dozen copies of what was essentially the same app, but that required SQL Server 2016)
Also, when I brought up postgres for a new project & mentioned it being open source, response was "that means it could vanish any day"
This is a very good and fair question. Postgres is growing a lot, but is it growing enough, according to its true potential? It has everything: incredibly robust and trusted; very large feature set; not under any company's direction; extremely liberal license. Should be conquering the database market, and by far is not!
I don't have an answer. I have potentially, many. Possibly I will blog about this at some time. But I believe that definitely it should be growing more and becoming more relevant than it is right now.
This is no detriment at all to all the fantastic work done by everybody; but just the ambition that Postgres can and should go farther.
Postgres never gave me the ability to quickly and easily build stored procedures that return multiple heterogeneous resultsets with ease. In SQL Server you do it like this:
CREATE PROCEDURE get_customer_and_orders
@id int
BEGIN
SELECT id, first_name, last_name, email, etc FROM customers WHERE id = @id;
SELECT id, store_id, created_at, etc FROM customer_orders WHERE customer_id = @id;
END
I've quickly built entire applications with this tactic as the centerpiece. You can argue that it moves business logic into the database layer and to that I'd say "good", at least for apps that are maintained by IT departments with many strong SQL people and not so many developers. If you know TSQL, you know that you can also do branching, looping and other logic operations within this same procedure - you can even decide to send back 3 resultsets instead of 2 if you want to and the client API allows you to handle whatever it received quite elegantly.
I think that features like this are why many businesses will stay on SQL Server. Also the high quality of tools for SQL Server that have no match in Postgres such as SQL Server Management Studio, SQL Server Data Tools, SQL Server Profiler, SQL Server Integration Services among many other such tools that are extremely well integrated.
This isn't meant to detract from your point, but just a fun anecdote -- my first job out of college was working on a product that had its entire business logic layer built in stored procedures. All of it. Hundreds and hundreds of lines per procedure. It "worked", but man was maintenance a bear. In a way, it was good; ever since I've been very comfortable with SQL. But I can't recommend it.
My limited experience with smaller companies (and now local government) shares this sentiment. For “small” databases (say, not hundreds or more of terabytes) I’ve experienced using stored procedures as a core of the ETL process.
That being said, haven’t some recent versions of Postgres added support for stored procedures or some variant? I’m curious if we’d seen any changes in performance if we experimented with switching over.
The example you gave makes no sense. It would be better expressed as a join. There may be examples where it makes sense, but I can't think of any. In SQL Server, I think of a stored proc as a logical operation that takes action and returns one object. If I want sub-structure within that object, I have the stored proc return one json or xml object.
The example given allows the application layer to instantiate a customer object with a list of order objects in one database query. Yes, you could do that by returning JSON or XML, but why have the extra step of parsing the text? Directly accessing the result set and acting on the object instances will be more efficient.
No it's not. A join means you have to wastefully repeat the same parent fields for every row. That's wasted cycles serializing and deserializing and it's more work on the client.
It's absolutely ridiculous to make a blanket statement about this very useful feature.
I've done the benchmarking on many applications of this tactic. It very often works better with multiple select statements. I'll certainly trust hard performance numbers against an ill conceived opinion any day.
And yet, MARS is not great for troubleshooting or testing (for database people writing SQL) and I have worked at many places where the complexity of many sets doesn't really justify not making two procedures.
Perhaps I'm interpreting your code incorrectly, but this seems like you can do this with PG as well? You just define a custom type and use that type in the "returning" clause? You can easily define a bunch of scalars (customer_id, first_name, ...) and an array (customer_orders[]) inside this type.
That is one way to do it, kinda sucks compared to just returning multiple result sets because you have the overhead of maintaining those custom types for every type of query you're going to need. I don't personally like that solution much for complicated apps which will have a ton of queries written against their database.
I see that as an advantage. What you're really doing is defining the interface for that function, and once the function is in use, you can't just change it willy nilly. Having to alter the type is in some way a "are you sure you want to do that?" layer.
It would be extremely nice if you could just specify that directly in the function and not have that sprawl where it's unnecessary. For a single result set you do exactly that with PG, just wish it would allow you to specify multiple.
As a contractor, I'm glad that MySQL is still around. It's an extremely useful red flag for potential clients.
It is almost guaranteed that if a client is running on MySQL, you can be certain their entire code base is going to be plagued with less-than-best-practices. Not to be too harsh, but MySQL's most prominent use case is for projects that start with "let's just stand-up a DB real quick and we'll sort out the hard stuff later".
I think a lot of people just don't know the new capabilities of PostgreSQL. PostgreSQL's competition isn't MySQL, its the new hybrid DBs like CockroachDb.
DBAs and Engineers that know MySQL will always use MySQL. In corps MySQL is a safe choice (sadly).
MySQL is a bit simpler even though pgadmin is a wonderful tool for learning postgres advantages. For me the difference became clear when needing to plot locations on a globe. PostGIS is super helpful
MySQL has way better UX for command line users. Postgres can't get away from legacy Ingres design, with a postgres user and a horrible command line client. Also, database/schema/tables hierarchy is a penalty over database/tables in MySQL.
I fail to see how Postgres' hierarchy is any worse than MySQL. You have one more layer of hierarchy which you do not need to consume, with the added feature of atomic renaming any of the layers.
If I'd compare this to MySQL, where you must move tables one-by-one to another schema (nay database) with the new name to 'rename' your database [0], then I'm very happy with postgres' hierarchy structure.
psql is easily one of my favorite tools. What makes the CLI "horrible" in your mind?
Also, database/schema/tables is great when doing things like multi-tenancy where you split customers by schema (which also makes horizontal scaling super easy if needed).
People go with what they know, the the LAMP stack was a classic for multiple years. And even today PostgreSQL's HA tooling isn't that great ( one can set up MySQL/MariaDB in active/active in a few minutes, and that's impossible with PgSQL, and even active/passive with easy failover requires more work and oftentimes external tooling).
It's not about needing it, it's about the security of knowing that you have proper redundancy that will work if you need it ( e.g. one goes down) without manual intervention.
In my little world, MySQL was popular because we could get enterprise level support directly from MySQL. It was the default choice from that point on. I don't know if that has changed. AWS changed that pretty quickly.
In terms of overall adoption, PostgreSQL doesn't need to catch Oracle or MySQL. The people that enjoy legacy platforms are not going to change (necessarily).
Real work gets done with PostgreSQL. You can't buy something from Amazon.com without using PostgreSQL. The more Amazon talks about that in their ecosystem, the more PostgreSQL becomes the default choice for all new developers within that (very large) ecosystem.
When you combine that with the loud presence of another 800lb Gorilla promoting PostgreSQL (MSFT), the market has spoken.
> I don’t have an MBA, but this to me is a clear case of The Innovator’s Dilemma. PostgreSQL protocol v3 is the incumbent, and protocol v4 and/or other protocols like HTTP are the potential disruptive innovators.
The fifth qualifier "New firm's business model differs significantly from incumbent" seems important.
The rest feel like Just So stories. Especially "Originate in low-end (less demanding customers) or new market (where none existed) footholds".
I think time has shown Peter Drucker to be more right. First target affluent customers and then ride the cost curve down towards market success.
I just reread Innovator's Dilemma, and searched the rest. Nothing about cost curves.
I've been trying to think of how "disruption" would apply to Tesla vehicles and Apple Silicon. Neither products came in as "toys".
Rather, I think both initially targeted very high end niches (novel use cases), and the underlying enabling technology was just starting their cost trajectories, whereas the incumbent mass market products were at the end of their trajectories. Plough metric tons of cash into R&D and wait for the new to overtake the old.
--
Back to AWS Babelfish.
AWS adds TDS and T-SQL compatibility to steal business from Microsoft Azure. Just some more bullet list items for middle management purchase bots to check off.
It'll surely improve, but switchover costs for non-trivial projects would still be pretty high. Back when I was warehousing medical records on both MSSQL and Oracle, all the hard work was tweaking the physical design, eg codesigning the indices and queries, to make stuff performant. I'm sure that's still true.
I think adding these features to Postgres is good for everyone but Microsoft.
--
I guess I object to the common use of "disruption". As though there's an element of surprise.
I'd suggest the OP framed AWS Babelfish in terms of competitive threats (aka insurmountable opportunities). And maybe predict some unforeseen consequences.
On the positive side: The meaning of "disruptive innovation" wonderfully Agile. Perfect for consultants, pundits, and polywogs.
