What I Wish I Knew as a New DBA: Your Questions, Answered - Hari Kiran

Last Friday’s webinar, “What I Wish I Knew as a New DBA: as told by veterans,” got more questions than we had time for. That’s a good sign: curious DBAs become good DBAs. So grab your coffee (or your third coffee, no judgment), and let’s work through them.

One housekeeping note: two of you asked almost exactly the same question about “everything looks normal but users are complaining.” Great minds think alike, so I’ve answered it once below.

1. How can a junior DBA build the confidence to challenge a developer or senior engineer when a query or change looks risky?

Here’s the secret: you don’t need to challenge the person. You need to question the change. That’s much easier, and much more effective.

Start with curiosity rather than confrontation. “Help me understand how this behaves on the orders table with 400 million rows” lands far better than “This will break production.” Questions invite people to think with you, and very often the developer spots the problem themselves while answering.

Then bring evidence:

Run EXPLAIN (ANALYZE, BUFFERS) on a staging copy.
Check what locks an ALTER TABLE will take.
Estimate how long an index build will run.

A senior engineer may overrule your opinion, but it’s much harder to overrule a query plan showing a sequential scan across the whole table.

Offer an alternative, not just a red flag.

“Could we use CREATE INDEX CONCURRENTLY instead?”

“Can we batch this update in chunks of 10,000?”

turns you from a blocker into a collaborator. And always ask the question every veteran asks:

“What’s the rollback plan?”

Nobody is offended by that one.

Finally, remember that protecting the database is your job. The team hired a DBA precisely so someone would raise these concerns. Speaking up isn’t overstepping; staying quiet would be.

2. I have a production issue and I don’t know the answer. How do I communicate that without losing the team’s confidence?

Teams don’t lose confidence in people who say “I don’t know yet.” They lose confidence in people who go silent, guess wildly, or pretend.

The magic formula is: what we know, what we’re doing, and when you’ll hear from me next. For example:

“Checkout queries started slowing at 10:42. I’ve ruled out CPU and connection limits. I’m now checking lock contention and recent deployments. I’ll update you by 11:15, sooner if I find something.”

Notice what that message does. It shows you’re methodical, it gives people something concrete, and it promises a next update, which stops the anxious “any news?” pings. Then keep that promise, even if the update is “still investigating, here’s what I’ve ruled out.”

And escalate early. Pulling in a senior colleague after fifteen minutes isn’t weakness; it’s good incident management. Veterans don’t know everything either. They’ve just become very comfortable saying “let’s find out.”

3. There’s so much to learn in PostgreSQL. How do I decide what to know deeply versus what to learn when needed?

A good rule of thumb: learn deeply whatever can lose/corrupt data or take down production. Learn on demand whatever makes things nicer, faster, or fancier.

Your “know it cold” list should include backup and recovery (including actually restoring, and point-in-time recovery), MVCC and how vacuum and autovacuum work, WAL and checkpoints, locking behaviour, reading EXPLAIN output and indexing fundamentals, roles and permissions, and replication basics. These come up at 2 a.m., when there’s no time to read the docs from scratch.

Your “learn when the situation calls for it” list includes things like specific extensions, foreign data wrappers, advanced partitioning strategies, logical replication edge cases, and the finer tuning parameters. It’s enough to know they exist and roughly what problem they solve, so you know where to look when the day comes.

Think of it as being T-shaped: broad awareness across the ecosystem, with a deep foundation in the parts that keep data safe and systems running.

4. What separates someone who knows PostgreSQL technically from someone who is actually good at being a DBA?

Knowing PostgreSQL is knowing how the engine works. Being a good DBA is knowing how the engine fits into a business, a team, and a very bad Friday.

A great DBA tests backups before they’re needed rather than assuming they work. They stay calm when everyone else is panicking, and that calm is contagious. They communicate risk in language managers understand (“this migration could lock the payments table for about 20 minutes during business hours”) rather than burying people in jargon. They write things down: runbooks, change notes, post-incident reviews. And they’re quietly paranoid in the most useful way, always asking “what could go wrong, and how would we recover?”

Technical skill gets you through the door. Judgment, communication, and ownership are what make people say, “We need that DBA on this project.”

**5. I accidentally deleted the data directory with rm -rf * and have no backups. Is there any way to recover it?**

First, a deep breath. Nearly every veteran has a story like this, even if theirs didn’t go all the way. Now, honestly: the odds of full recovery are low, but not zero, and what you do in the next few minutes matters a lot.

If PostgreSQL is still running, do not stop or restart it. On Linux, deleted files that are still held open by a process continue to exist on disk until that process closes them. Look under /proc/<pid>/fd/ for PostgreSQL’s processes and you may see entries marked (deleted). Those can sometimes be copied out to another disk. This rarely gives you a complete, consistent cluster, but it may rescue some files. Get experienced help before relying on what you salvage.

Stop all writes to that filesystem immediately. Every new write can overwrite the blocks where your deleted data still physically sits. If possible, unmount it or remount it read-only, and never write recovery output back onto the same disk.

Try filesystem-level recovery. Results vary a lot by filesystem; XFS in particular is much harder. Work from a disk image rather than the original if you can.

And once the dust settles, turn this into the most valuable lesson of your career: set up automated backups with a tool like pgBackRest or Barman, archive your WAL, and, most importantly, practise restoring regularly. A backup you’ve never restored is just a hopeful rumour.

6. Everything looks normal (CPU, memory, connections, queries) but users are still complaining. Where should a junior DBA start?

Ah, the classic mystery! Averages are wonderful liars. Your dashboard says “fine” while one in fifty users is having a miserable time.

Start by getting specific about the complaint. Which users? Which screen or feature? Since when? All the time or at certain hours? “The app is slow” is a feeling; “saving an invoice takes 12 seconds since Monday afternoon” is a lead.

Then look at what’s waiting rather than what’s working. A system can have low CPU while sessions sit blocked. Check pg_stat_activity for wait_event values and sessions stuck in idle in transaction, and look at pg_locks for blocking chains. One forgotten open transaction can quietly hold up everything behind it.

Look at tail latency, not just averages. In pg_stat_statements, compare mean_exec_time with max_exec_time and stddev_exec_time. A query that’s usually 5 ms but sometimes 8 seconds tells a story. A plan change after an ANALYZE or a data growth spurt is a common culprit, and auto_explain can catch it in the act.

Check what sits between the user and the database. If you use PgBouncer, clients may be queueing for a connection (look at cl_waiting in SHOW POOLS) while the database itself looks relaxed. Network latency, DNS, load balancers, and the application’s own connection pool are all fair game.

Check storage latency rather than just throughput. iostat -x showing high await times can make everything sluggish while CPU stays calm. Also look for checkpoint spikes, heavy temp file usage, table bloat, and autovacuum struggling to keep up. If some reads go to a replica, check replication lag too; users may be seeing stale data and calling it “broken.”

And sometimes, after all that, the answer is: it isn’t the database. That’s a perfectly good finding. Showing evidence that the database is healthy helps the team look in the right place, which is a real contribution.

7. How do you handle a client who isn’t receptive to other people’s decisions or recommendations?

This one is as much about people as it is about PostgreSQL, and it’s worth getting good at.

Start by understanding what’s behind the resistance. Often it isn’t stubbornness but a concern nobody has addressed: cost, downtime, a bad past experience, or fear of change. Ask what worries them about the recommendation, then listen properly.

Speak their language. Instead of “we need to upgrade from an end-of-life version,” try “the current version no longer receives security fixes, which puts customer data at risk and may matter for your compliance audit.” Tie technical recommendations to things they already care about.

Offer options rather than ultimatums. “Here’s the ideal fix, here’s a lower-cost middle ground, and here are the risks of doing nothing” gives them control, and people accept decisions more readily when they’ve had a hand in shaping them.

Finally, if they still decide against your advice, respect that it’s their call, but document it clearly and politely: what you recommended, what risks you identified, and what was decided. That protects everyone, and it often means that when the risk does show up, they’ll come back to you with a lot more trust.

Keep the questions coming!

Thank you to everyone who joined and asked such thoughtful questions. If there’s one thread running through all of these answers, it’s this: great DBAs aren’t the ones who never make mistakes or always know the answer. They’re the ones who stay curious, communicate clearly, protect the data, and keep learning.

Got a follow-up, a war story, or a question we didn’t cover? Do let us know. We’d love to keep the conversation going. And please, before you do anything else today, go check that your backups are actually restored.