Too Good for my own damn self!
I had spent the last couple of years on my thumbs resting on my laurels. From initially being on the cutting edge of MariaDB technology through the use of single-source multi-primary replication (where every slave is a master, and every master a slave: kind of like a 3D replication) in a multi-datacenter, high availability production setting; being one of the first to realize the UINT32 GTID bug; to 4-9’s system stability for the past 4 years: I left employment realizing that the world had changed, the technology was advancing, and I hadn’t graduated from Ubuntu 14.04, whose end of life was in April of 2019.
What is going on out there?
The requirements on the job boards were changing, and it wasn’t enough that I had built a database architecture capable of growing a company from $2 mil in revenue to $120 mil in 6 years. It’s not the dollar amount that I’m hyping: it’s the number of transactions to get to that dollar amount that matters. Everyone from small startups to the Fortune 500 were wanting to move to the cloud. Developers wanted to containerize.
The Illuminati plan for world devops domination …
I felt like there was a shadow influence underneath it all with a single and deliberate task: to get rid of sysadmins and DBA’s.
Well – not really . . .
Managed systems would still be handled by sysadmins. But just a smaller amount of them. Your development team wouldn’t need to have one if you were managed by Amazon or Google in the cloud. The providers would handle the hardware, backups, network, regions and availability zones. If you had a smart developer with some sysadmin skills in the background to connect your development to the cloud, you wouldn’t need your own sysadmin. Well, if you had a smart developer with good google-fu, then sysadmin background wouldn’t be needed either.
But can you SQL Query? I mean, like, query sql?
The number one question I was asked through a list of interviews: How is your sql query tuning?. . . Ummmm. You mean performance tuning through configuration, and an understanding of how a query flows through the optimizer, decides on paging and the creation of temporary space from whatever is the underlying storage, and finally processing and producing the result set? Yeah – I got that . . . Nope. Sql Query Tuning. Developers write queries, but sometimes they aren’t very good. We don’t need an operational DBA, we just need someone to check on what they are doing. I had been approached by several DBA consulting groups who were looking for MySQL experience in the cloud. None of them were looking for the ability to configure hardware. DBA’s were clustering into consultant groups, which makes sense. Being a lonely and singular DBA in an enterprise level operations team sucks for 24/7 on-call. It seriously makes a 6 year employment seem a lot longer.
I get why it’s happening.
We tried to hire additional MySQL DBA’s in South Florida, but unfortunately, there were no viable MySQL DBA’s. Four out of the six years at my former place of employment, we couldn’t find a dedicated MySQL DBA with enough skill to handle the job on day 180, much less day 1. While attempting to train a developer to DBA, I couldn’t do it. He just didn’t have the ability to understand the viewpoint a DBA must have in order to be good at the job. For those four years, I was the sole DBA. For the other two years, I might as well have been the sole DBA, but at least I was on an on-call rotation (those 4-9’s helped!). Prior to leaving, we hired a person who had the title of MySQL DBA at his previous job, whose skill set was on the same level as job operator. Commence dba training…
I began to understand that outside of New York, Dallas, Silicon Valley and Los Angeles – there was probably a drought of experience in database systems, understanding the internals of MySQL, Query construction. At the same time, development and the need for data continued. LAMP/LEMP, AWS RDS and Google Cloud used MySQL to start, and every programming book since at least 1997 (memory disclaimer below) included a section on how to create a database using MySQL. Developers required it, but did not necessarily have the ability to think like an optimizer – so even with 400GB of RAM and 16000 IOPS, their queries were slow. But there were no viable DBA’s, so the next best thing was a managed system, with a short term consultant.
Maybe, that’s the future of MySQL expertise in a cloud enabled world. That is precisely what I’m about to find out.
Disclaimer: Ok – the first PHP book I ever owned was published in 2000. It was the Wrox Programmer to Programmer: Beginning Php 4. by Wrox Author Team Staff (Choi, Kent, Lea, Prasad and Ullman), Copyright 2000. I’m not 100% sure if there were any books from 1997 that included MySQL, but the above book is what really started me in this career. My first PHP/MySQL program was written in 1997. I may have just figured it out on my own. –Ed
What do the employers want?
- On-Prem DBA’s. There’s still pushback on remote DBA’s for many organizations. Seriously, a DBA’s most important work happens at home, on VPN, attempting to ensure revenue continues to flow while the developers are all asleep. If they can do that at home, they can answer a question on Skype during the business day on query construction.
- Best approach to Cloud Solutions: AWS RDS, CloudSQL, Azure and others. One of the topics I’m actively studying.
- Postgresql and Oracle or MS SQL Server. There seems to be an assumption that if you know MySQL, you should also know Postgresql and Oracle or SQL Server.
- Clustering via Galera or InnoDB Cluster. Seems like a lot of organizations have tried it and it just didn’t work right.
- Some No-SQL (ie. MongoDB).
- Some Memcache (Redis comes up a lot).
- Configuration Management (Puppet, Chef, Ansible). This had always been a sysadmin task. . . sorry sysadmins.
- MySQL Hi Availability tools (MHA, MRM). Not something that I needed to use, but some people think it’s necessary.
- Expert proficiency in at least one programming language: Python comes up a lot. PHP in many web based organizations.
- Bash. I believe you should definitely have some bash scripting skills.
- Jenkins for database provisioning and deployment. I’m going to look into this in the future.
- Docker and Kubernetes. Seems everywhere to me. I am actively reviewing containerized MySQL’s and Galera to see what the hype is about. Even ClickHouse is working on a Kubernetes operator for ClickHouse clusters.
- Percona Tools. It may not be on the job listing, but someone will ask you about it. The one I always forget is pt-query-digest. Set slowlog long_query_time to 0 and have fun.
What I’m interested in, but not seeing:
- Maxscale and MySQL Router. Okay – MariaDB and Percona recruiters are interested in it, but I’m not seeing it as a requirement for MySQL DBA’s.
- Columnstore. MariaDB has been talking a lot about their ColumnStore engine. Although it’s not a “MySQL” database, I have been looking at ClickHouse for audit and slow log analysis, as well as a backend for Zabbix monitoring. Percona also likes ClickHouse, but, well, Zaitsev brothers ….
- Zabbix Monitoring. I know there are other monitoring solutions out there, but I’ve really been looking at Zabbix for all phases of monitoring for on-prem DBA’s, and possibly for cloud via the cloud subscriptions.
DBA Pay
- Pay for salary seems to be between $80K to about $120K. Unfortunately, I also believe that good MySQL DBA’s are underpaid (supply vs demand). I would also temper that statement by saying that many MySQL DBA candidates are not dedicated (came from a different RDBMS or from development/sysadmins) nor experienced, which leads to the skew in perception of quality for MySQL DBA’s.
- Hourly contract runs the gamut from $60 to $125, depending on many factors. Consulting group contracts try to stick to $60 – $75 ….
