How to Buy a Penetration Test and Not an Automated Scan

Three proposals for the same web application. One lands well under five thousand dollars, one around fifteen, one close to forty. All three say “penetration testing,” promise a report, a retest and a named consultant, and explain nothing. Those figures are illustrative; the spread is not. The documents describe different work and none of them admits it.

Which is why “who should we use?” is the wrong question to ask a forum. Quality varies enormously inside any price band. Two-person shops produce the best security document some engineering teams have ever read; large well-known firms sell scans with a cover page. A name from a thread tells you what one buyer got, on one engagement, from one tester. Testers move.

Most disappointing penetration tests are automated scans sold as manual testing. The way to avoid buying one is to specify the testing methodology, the tester’s time and the report format before you ask anybody for a price. Write that document and the shortlist sorts itself, because a good share of the vendors you approach will not commit to it and will say so on the first call.

Four different things get sold as a penetration test

Not quality tiers. Separate products, and a vendor can be excellent at one while being the wrong purchase for you.

Vulnerability scanning. Automated signature and version matching: missing patches, known CVEs, expired certificates, weak TLS, exposed services, default credentials. NIST’s technical guide states the limit plainly: scanning finds surface vulnerabilities and cannot reach what only appears through combinations of attack patterns. Effort is machine time plus a human clearing false positives. Own it and run it continuously rather than buying it annually.
Automated or continuous pen testing platforms. Scanning plus scripted exploitation and lateral movement, usually with an agent inside. Genuinely useful on internal networks and Active Directory, where they chain credential reuse and misconfiguration into a real attack path and prove it. They cannot reason about your business rules. Nobody wrote a module for your invoice approval workflow.
Manual penetration testing. A human reads your application, forms hypotheses about how it can be abused, and tests them. This is where the findings that matter come from: one tenant reading another’s records, a support user escalating to admin through an impersonation feature, an order that completes when the payment step is skipped. NIST’s framing is that penetration testing corroborates whether a vulnerability is genuinely exploitable and shows how lower-risk issues aggregate into real risk. The unit is tester days, and a moderately complex application runs to a working week or more of one person’s time before reporting starts.
Red teaming. Objective-based adversary emulation run against your detection and response rather than an asset list. The output is whether your SOC would have caught it and how long that took, not a vulnerability inventory. Never had a pen test? Wrong first buy: you fund a stealth exercise to learn things an authenticated test hands you in a week.

You can usually tell which one a proposal sells. The tells: the methodology section names a tool and little else; pricing is per IP or per URL rather than per day; turnaround is quoted in hours; sample severities are untouched tool defaults and every finding is a version number; you got a fixed price from a one-line description with no scoping call, so nothing was scoped and nothing estimated. Strongest signal of all: nobody asked you for credentials or test accounts.

Name the methodology so the proposals become comparable

RFPs that say “industry standard penetration testing” have no content. Every vendor answers yes, and you compare prices on undefined work. Name a published methodology instead.

For web applications and services that is the OWASP Web Security Testing Guide, currently version 4.2, published 3 December 2020. Its procurement value is the identifier scheme: every test scenario carries an ID of the form WSTG-<category>-<number>, and OWASP recommends carrying the version too, so a test reads as WSTG-v42-INFO-02. That is coverage a proposal cannot wriggle out of, and it lets you require gaps to be stated rather than silently skipped.

For the process around whichever technical guide applies, NIST SP 800-115, Technical Guide to Information Security Testing and Assessment (September 2008) still sets the shape: a four-stage methodology whose attack phase loops back into discovery as the tester learns more. Its tooling references are ancient. Its process framing is what a serious methodology section still looks like.

Naming a standard does not make anyone follow it, so make the coverage mapping a contractual deliverable, and match the reference to the asset. A vendor offering WSTG coverage for an internal network assessment has not read it.

Scope decides what the test can possibly find

Authenticated testing on real roles is the whole game, and it is the thing cheap tests omit. An unauthenticated test of a SaaS product finds the login page, some headers and a TLS configuration. Broken access control requires being logged in to find, and that is where the serious modern bugs live. A report with no authenticated findings usually means the tester never got inside, not that nothing was wrong.

So put the accounts in the scope document:

One per role, written out: super admin, organisation admin, standard user, read-only, billing, support or impersonation, an API-only service account, and the pending-invite state everyone forgets.
Two in the same role, so horizontal access control can be tested at all.
Two separate tenants, so cross-tenant isolation can be tested. On a multi-tenant product this is the highest-value item in the engagement. Arrange MFA on every test account in advance; half a day lost to a locked account is half a day you paid for.

Then decide how much to hand over. Withholding documentation to keep the test “realistic” buys realism you do not need and burns paid days on reconnaissance a real attacker would happily spend months on. Give them the enumerated asset list, the architecture diagram, the roles and permissions matrix, the stack and the OpenAPI specification. If you can give read access to the source, do it: a tester reading your authorisation middleware for an hour finds things no black-box probing surfaces.

On APIs, the interface is a client and the endpoints behind it are reachable without it. Ask whether coverage means every endpoint in the specification or only those the UI calls, because those are very different amounts of work. And ask about old versions: the v1 you deprecated when v2 shipped is often still deployed, still authenticating, and missing two years of authorisation fixes.

What has to be true before anyone starts

Written authorisation signed by someone with actual authority, rules of engagement covering permitted techniques and testing windows, an out-of-scope list, agreed stop conditions, a contact path that reaches a human at 3am, your cloud provider’s testing policy, and a written agreement on how evidence is stored and destroyed. All of it belongs in the contract, not an email thread. We covered building it, including the AWS, Azure and Google Cloud positions, in how to run a pen test without taking production down.

Tester days are what you are actually buying

One question separates a testing firm from a reseller of scans: how many days of manual testing effort, by how many people, at what seniority, and how is that time split between reconnaissance, exploitation and reporting?

Ask it in those words and listen to the shape of the answer, not the number. A real firm answers immediately and in specifics, because tester days is how they cost every job. Something like: eight days, one senior tester plus two days of a second on the API, two days mapping and account setup, four days testing, two days writing and internal review. A reseller cannot produce that breakdown and redirects to the comprehensiveness of their methodology.

Then three follow-ups:

Will the named person do the work or review it? The CV in the proposal and the human at the keyboard are often different people. Name the assigned tester in the contract, with a right to be told if they change.
How many reporting days, and who reviews findings before the report ships? Writing findings properly is real effort, and a technical QA pass is the difference between a document engineers can work from and a week spent arguing about false positives. A proposal with no reporting time is telling you the report is generated.
What would you cut if I halved the budget? The honesty test. A good tester names the asset to drop and what you will consequently not know. A reseller keeps the scope identical and quietly reduces depth, the worst outcome available to you, because the report looks identical.

What certifications evidence, and what they don’t

Individual offensive certifications evidence hands-on ability under controlled conditions. OffSec’s OSCP, the best known, is earned through a proctored 24-hour practical exam compromising live lab systems, plus a further 24 hours to submit a penetration testing report. The holder has proven they can find and exploit vulnerabilities against the clock and write it up. It does not evidence depth on a complex multi-tenant application, cloud control-plane work, or the judgement that makes a report useful. A floor, and a meaningful one.

Company-level schemes work differently. CREST accredits organisations across service categories including Security Testing, Red Teaming, Security Operations Centres, Incident Management and Threat Intelligence, against published standards that include a general company requirements standard. Know what that covers: CREST states there is no mandatory requirement for individuals to be CREST qualified for a company to achieve accreditation, and that individuals are instead assessed on their skills and experience or any qualifications they hold. So accreditation evidences that the firm’s processes, methodologies and staffing were assessed against a published standard by someone other than its own marketing department. In a market with no licensing that is valuable. It says nothing about your assigned tester.

The honest limit: certification says a vendor clears a floor. It ranks nobody and predicts nothing about your report. What predicts your report is which individual gets assigned, how many days they get, and how good your scope document was.

The deliverable specification buyers almost never write

Buyers spend weeks on scope and then accept whatever report arrives. Specify the deliverable instead. Seven requirements and one test:

An executive summary saying what an attacker could achieve, not a count of highs and mediums. Two pages a non-technical reader can act on, stating what was tested and what was not.
Findings with reproduction steps and evidence. Affected asset, steps a developer can actually follow, the request and response or a screenshot, and impact in your business’s terms. A finding nobody can reproduce becomes a six-week argument.
Severity with a stated scoring method, including the version. If it is CVSS, which version, and whether environmental metrics were applied or only base scores. A base score is context-free: it does not know one affected host is internet-facing and the other sits in a lab. Require the method in writing, plus a separate view of exploitability in your environment.
Remediation specific to your stack. “Enforce server-side authorisation in this middleware” beats “follow secure coding practices.” You enable that by handing over the stack at scoping, so ask for it and hold them to it.
A retest in the original price, within a defined window. Sixty or ninety days from delivery, re-verifying every finding you claim fixed, delivered as a delta report. If the retest is a separate purchase, remediation becomes a second procurement cycle and half your findings stay open.
A machine-readable findings export. CSV or JSON with stable IDs, severities and affected assets, so results land in Jira or GitHub Issues without anyone retyping a PDF. Ask what formats they produce before you sign.
A debrief call with the tester, plus a route to ask them questions during remediation.

Then the test: ask for a redacted sample report before signing, and read it critically. Can you follow the reproduction steps? Is there evidence, or assertion? Is any single finding a business logic or access control flaw, or is every one a version number and a missing header? Does the remediation name a real technology? A vendor who will not show you a redacted sample has answered the question.

What to do with the report, where the value usually leaks

The engagement ends and the PDF goes into a folder, the most common way organisations waste a good test.

Triage by exploitability, not the severity column. A medium that is internet-facing, unauthenticated and reachable by anyone with a browser outranks a high needing domain admin you already control tightly. Ask the tester directly at the debrief: if you had one week inside us as an attacker, which five of these would you use? That ranking beats the scores.

Move every finding into the tracker your engineers already use, with a named owner and a date. This is what the machine-readable export is for. Findings that live only in the report do not get fixed, and at the retest you explain why.

Look for classes, not instances. Three missing authorisation checks on three endpoints is one design problem, and patching the three named endpoints guarantees you buy the same report next year. Fix the pattern: centralised authorisation, a test that walks every endpoint as every role, a checklist item at design review. Then take the recurring classes back to the people writing the code, with the requests the tester actually used.

Book the retest before remediation starts. The window you negotiated otherwise expires quietly during a busy quarter, and you pay again for something you already bought.

Does the vendor have to be US-based?

Sometimes, for checkable reasons rather than by default. The ones that genuinely bind: a customer contract or regulatory position restricting where your data and the test evidence may be stored and who may access them; personnel screening you need to verify; time zone overlap, because the daily check-in and the stop-the-test call both need someone awake; and report writing you can hand an auditor without editing. Otherwise a strong tester elsewhere beats a mediocre one nearby. Put data residency, background checks and working hours in the RFP and let vendors answer, instead of using location as a proxy for quality.

Do this and the original question dissolves. Once you have written down which of the four products you want, the methodology, the accounts and roles the tester gets, the manual day count and the deliverable spec, the proposals become comparable and the weak vendors select themselves out.

If you are turning this into an RFP, a scope document and a deliverable specification, that is work AB7 Solutions does, and we do the testing itself. Our cybersecurity practice covers VAPT and manual penetration testing across web applications, APIs and internal networks, firewall and server security review, and SOC monitoring for the detection side of the same estate. The reason to call us on this problem is the judgement this article is made of: we will tell you which of the four products you need, and we will say when the honest answer is a continuous scanning capability you run yourself rather than an annual engagement from anybody, us included. Send the proposals you already have and we will mark up the day counts, the authenticated coverage and the deliverable gaps. Call +1 321 341 7733, email ab@ab7solutions.com or director@ab7solutions.com, or start at www.ab7solutions.com.

Sources: OWASP, Web Security Testing Guide v4.2 (3 December 2020); NIST, SP 800-115, Technical Guide to Information Security Testing and Assessment (September 2008); CREST accreditation standards; OffSec, PEN-200 / OSCP exam format.