This is a guest blog post from Mohit, who is working on re-writing the web interface for OWTF from the ground up in React!

He is sophomore at the International Institute of Information Technology, Hyderabad (IIIT-H), majoring in Computer science and Engineering. His project will have a tremendous impact on the usability and ease of use for OWTF.

And with that, a big welcome to Mohit!

A word of thanks

Before I go any further, I would like to express my gratitude to Google, my mentors, family members, lecturers, friends and to all those who made it possible. Thank you very much :)

OWASP OWTF-Web Interface Enhancements

Mentors: Viyat Bhalodia, Abraham Aranguren, Anshul Singhal

The project aims to refactor and re-write the UI for OWTF from a mix of Jinja/Tornado templates (a popular Python templating engine) and React to stable ReactJs based interface. The project will also introduce new features to the web interface — including a new page layouts and styling along with modification to the build system.

Check my GSoC proposal for more details on design decisions and implementation details.

Community bonding period (April 23 2018 — May 13 2018)

In the community bonding period, the student is expected to get familiarize with the community and the mentors — and get familiar with community practices and processes, work on project related issues, go through the documentation, have a talk with the mentors on refining the project and get a deeper understanding of the project and to know what exactly needs to be done during the whole course of time.

My project focuses on refactoring the UI of the OWTF web application from the mixed Jinja/Tornado templates to a stable, fully ReactJs based interface with Redux as its state manager. Since I haven’t used much of React before, I began by looking at some demo React-Redux projects to get a deeper understanding of React and get familiar with all the terms related to React. These are some of the links that helped me getting started with React-Redux:

https://medium.com/@rajaraodv/step-by-step-guide-to-building-react-redux-apps-using-mocks-48ca0f47f9a

https://github.com/buckyroberts/React-Redux-Boilerplate

An initial base for the project had already been created for the project. My mentors suggested to go through the code, understand the architecture and find/debug any errors that I find. In addition, I also looked at the old interface code to understand the functioning of each component on the application.

My first PR:

After all the reading, I made my first contribution by implementing the sessions functionality on the targets page. OWTF sessions are really an abstraction of a simple state system for OWTF that lets the user switch between projects and work seamlessly between different target groups. In the old implementation, the load session and change session functionality were already created — here are the steps I followed to port the functionality over to React:

• Wrote the actions for each event — Create session, Create session success, Create session error.
• Added the post session request handler using react sagas.
• Implemented reducer to manage the state after a session create.
• Finally, connected the Tornado API to the React frontend using mapDispatchToProps function.

For more details, here is the link to the pull request for this change.

Following are some of the links that I found useful while implementing the create session functionality — mostly focusing on how to use React-Redux-Sagas:

https://github.com/redux-saga/redux-saga

https://www.youtube.com/watch?v=msx0Qiu8NxQ&list=PLw7fHewFA6OTyUnLiZ1HQvYdzjp9ARMQw

That’s it on the community bonding period. Mohit will be posting weekly updates about the implementation details on the new interface — stay tuned!

My first memory of working in the open source community is the summer of 2014 — I participated in the Google Summer of Code with OWASP OWTF! Looking back, it was an eye-opening experience for me, a novice security student to work with brilliant mentors, and learn from people working on other ideas.

I won’t lie — working in open source security community has been a tremendous advantage for me in professional networking, new friends, finding new opportunities (speaking at BlackHat Arsenal!) and many more :).

Maximizing your chances of being accepted in GSoC

Here are some of observations that Abraham Aranguren (original OWTF Project author and lead) made during the previous Summer of Code years:

1. People with pre-GSoC project involvement are generally rank the highest.
2. People who started working on their proposals EARLY ranked the highest.
3. People who worked the hardest on their proposals ranked the highest

General tips on how to write a hard-to-refuse proposal!

1. Think about an idea to write up a proposal — consult the ideas page or the Github issue tracker for this (https://www.owasp.org/index.php/GSOC2018_Ideas and https://github.com/owtf/owtf/issues)
2. Take the proposal seriously, as if you were looking for a job:
You are proposing a solution, that is complete, solid and believable and you add references to demonstrate why you will do a good job implementing that idea.
3. Once you have a draft send it to your mentor, friends or the project leaders/maintainers for a review.
5. You improve the proposal from the review comments and send it for review again!
Repeat this iterative cycle until your proposal is concrete, clear and outlines your idea perfectly.

Here are some tips on how to write the actual proposal.

tl;dr To create a winning proposal: Have lots of images, a strong project plan, believable timelines, solid pre-implementation research, pre-GSoC project involvement, “hammering draft review cycles”.

Let the pre-implementation research drive your proposal:
How did other libraries/tools solve the problem you are trying to solve?
What were the challenges? How were the challenges solved? Based on that, how will you solve the problem? Are there online comparatives? (i.e.
accuracy, speed, reliability, benchmarks, etc. Use them generously!!)
If your proposal does not clearly answer these questions you will be in
the maybe zone.

Let me introduce what Heilmeier’s Catechism is:
A set of questions credited to Heilmeier that anyone proposing a
research project or product development effort should be able to answer
http://en.wikipedia.org/wiki/George_H._Heilmeier#Heilmeier.27s_Catechism

In the context of the GSoC, Abraham Aranguren (my mentor, to whom I am deeply indebted to, thanks Abe!) mapped the Heilmeier Catechism
questions to (potential) GSoC proposal sections.
Why? Google Summer of Code mentors will knowingly or unknowingly try to see if your proposal is able to answer the “nasty Heilmeier questions” ;).

IMPORTANT: Does your proposal answer these questions? if not, it
probably needs more work!

You should try to answer these questions indirectly in your proposal (i.e. DO NOT COPY-PASTE THESE QUESTIONS INTO YOUR PROPOSAL! ).

• Intro/Goals: What are you trying to do? Articulate your objectives using absolutely no jargon.
  Translation: Your introduction and even goals should be something that “even your parents/grandparents can understand”.

Who cares?
Translation: What people will benefit from your project?

If you’re successful, what difference will it make?
Translation: What are the advantages of your project?

• Pre-implementation research: How is it done today, and
  what are the limits of current practice?
  Translation: How do other libraries/tools currently solve the problem
  that you are trying to solve? What are the problems/limits they face?
  How do they try to mitigate those problems?

What’s new in your approach and why do you think it will be successful?
Translation: Based on your pre-implementation research, how will you solve the problem?
PRO TIP: If your proposal answers this in a solid scientific fashion,
you beat 90+% of your competition!

• Project Plan: What are the risks and the payoffs?
  Translation: What are the challenges and benefits of your project? How
  will you mitigate those challenges?

What are the midterm and final tests to check for success?
Translation: How will the mentors know if your project was successful?

• Project Plan/Timeline: How long will it take?
  Translation: Does your timeline make sense?

Your goal should be to make mentors/reviewers fall down their chairs when they see your proposal! :)

If you are active in OWASP or the general application security community and willing to mentor students for OWTF, please contact me! DMs open on Twitter :)

Some more excellent references

• http://www.di.ens.fr/~baghdadi/TXT_blog/5_advices_to_get_your_proposal_accepted.lyx.html
• https://wiki.illumos.org/display/illumos/How+to+write+an+excellent+GSoC+proposal
• http://seccoalegsoc.blogspot.com/2013/05/accepted-project-dynamically.html

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — OWASP Offensive Web Testing Framework (OWTF) is a great tool for organizing all information collected by the tools during a penetration test. The framework comes bundled with a more than 50 plugins which are simple Python files implementing a function with a set of commands (“resources”) and carefully chosen parameters . These plugins can be launched through a command-line or a web interface. Running a plugin generally involves invoking the specific tool for the test and storing the results in an organized directory structure for later analysis.

The plugin results are presented in an interactive report on the web interface. The image below shows a plugin “PTES-003” (corresponding to the PTES VNC test standard), which runs a command to test for weak VNC passwords given a host IP. This is one of OWTF’s auxiliary plugins — mostly using Metasploit to script complex tests.

OWTF interactive report includes many useful functionality including the possibility to rank the findings by severity, the information about when the commands were executed, write notes about, etc. We can make the report even better by parsing this command output and showing more information to the penetration tester!

One of the downsides of the current report is the way the results are stored (unstructured) and it could be modified to show relevant pieces of data from the command output by parsing the command output. For example, the result shown in the image above did find weak VNC credentials but it is not clearly distinguishable from the long command output. By parsing the command output of the plugin, we can make OWTF more intelligent by adding functionalities to answer human readable questions like

“How many vulnerabilities were found for the target?”, “How many of them were critical?” or “How many weak credentials were found in the session?”

Since OWTF can launch any command line tool by including a simple Bash script in the plugin file, the heterogeneous nature of the results makes it complicated to extract the important pieces of data from the results. OWASP DEF is an OWASP project whose purpose is

[…] define a simple, open format for exchanging data between pen test tools!

Although the idea sounds good (and very similar to the “failed” Mozilla Zest specification), there are but few (if any) popular security tools which allow that format. OWTF also has a sister project called PTP which tries to normalize the tools results into a global severity and ranking information. I have written a tool called unisecbarber, which does something similar, but is based on Faraday plugins.

Data Normalization

Unisecbarber can normalize and parse outputs from more than 60 different security tools. The basic flow of the tool is as follows — on receiving a command as input, it runs the command as a new process, parses the results and returns a JSON object of the normalized output data from the results. It is trivial to extend this to add support for a new tool by simply writing a new plugin for it.

For this OWTF module, I used the Unisecbarber tool to normalize the tool results. This module changes the OWTF plugin execution flow by adding an extra parsing step to the flow. In the sequence diagram shown below, all commands will pass trough unisecbarber first (if not supported, it will fallbacks to the original execution flow) to ensure we get normalized tool outputs. Once we get the normalized results from the command output, the data is persisted in the OWTF database backend (PostgreSQL).

Changes to the database schema

OWTF uses a relational database, PostgreSQL, to persist information. The following schema is used:

Using the normalised data from unisecbarber and the brand new database schema, OWTF can understand the results and answer what specific information was found.

For example, to answer the question “How many vulnerabilities were found for the target?”

OWTF would execute the following SQL query,

SELECT COUNT(DISTINCT v.id) FROM vulns v JOIN command_register_vuln crv ON crv.vuln_id = v.id JOIN command_register cr ON crv.command_register_id = cr.id JOIN plugin_outputs po ON cr.plugin_output_id = po.id WHERE po.target_id=1;

Results

The results from the module are presented in a nice report (the design is not finished yet). The new information can be viewed by command, plugin, target or the whole session. It looks like

As seen from the screenshot above, the credentials found by “PTES-003” are clearly presented in the credentials table.

Using this module, OWTF now has the ability to show clear actionable information to the user!. There is still a lot of work to do but the foundation for a future implementation of an intelligent plugin output parser and report UI.

Future work

Here is a short list of things that are on the roadmap:

• Implement an admin interface to edit findings.
• Implement a merge strategy to avoid duplicates.
• Improve the UI/UX of the reports
• Export the findings report to PDF / HTML .

Want to contribute? PRs welcome at https://github.com/csk/owtf!.

The project will be moved to the OWTF organization as a pluggable module soon! Stay tuned :)

Recent releases have been a small tribute to delicious Indian food, but especially dedicated to all those hard working Indian contributors who have continuously demonstrated their passion, professionalism, brainpower and incredible performance, without which OWTF would not be the awesome tool it is today. This release is named after all of you, thank you!

IMPORTANT: The support for 1.x releases has now ended and you should pull in the latest changes or download the latest release! Therefore, if you are coming from an old OWTF version, please run the following commands after downloading OWTF 2.0:
WARNING: This will delete everything in your OWTF database!

bash scripts/db_setup.sh clean

bash scripts/db_setup.sh init

New to OWTF? No problem!

Get it here! — https://owtf.github.io/#download :)

This release includes many stability and bug fixes. The entire codebase has been refactored to PEP8 (with some custom checks and modified requirements) standards.

New features

• A revamped installation process, using virtualenv.
• Moved all user configuration to the home directory.
• Added a Dockerfile to test OWTF on unsupported systems (macOS and Windows).

Bug fixes

• Removes old / unused / dead code.
• Lots of PEP-8 changes.
• Resolves an old proxy bug in e1ba544.
• Resolves many proxy SSL errors
• Fixed severity labels in the UI
• Improved helper scripts for setting up OWTF
• Fixed Debian installation scripts to point to Kali rolling.
• Fixed SIGINT errors in SSL testing scripts.
• Deprecate support for SamuraiWTF distribution.

View the full changelog here.

Downloads

• Source code (zip)
• Source code (tar.gz)

Originally published at blog.7-a.org on October 27, 2017.

bash scripts/db_setup.sh clean

bash scripts/db_setup.sh init

New to OWTF? No problem!

Get it here :)

https://owtf.github.io/#download

Release Notes

This release includes many new features and countless bug fixes. This release would not have been possible without the help of a number of pre-GSoC contributors, mentors, and everybody who sent us cool ideas, feedback or reported bugs. In particular, this release is dedicated to our Indian contributors without whom this release would not have been possible. As a wrapper tool that depends on many tools, migration from Kali 1.x to Kali 2.x was a little bumpy: this road saw more bugs fixes/reports from new contributors and users, occasional feature requests and countless fixes (that were long due) which made this release possible.

Important Features and fixes

• Kali 2.x support
• Functional tests suite included => build passing(!)
• Progress bar added to the web interface
• HTTPrint signatures updated
• Updated CMS Explorer lists
• Minimal auxiliary plugin support added back
• SSL Labs API integration
• Resolves SQLAlchemy deadlock and improved proxy handling
• Fixes all Metasploit plugin functionality
• General UI improvements
• CWE and OWASP Top 10 mappings
• Improved worker UI controls = adds Pause All, Resume All functionality
• Supports Debian-based distributions
• Target manager UI improvements = bulk delete/remove

Full Changelog

Implemented enhancements:

• xxx_testgroups.cfg should be moved to /profiles #670
• OWTF takes few steps to start #638
• Session Modal breaks for large session names #635
• Check for tools before running commands #632
• Adding Issue and Pull Request templates #599
• Debian and Samurai install scripts are not executable. #573
• Increase readability of manual installation output on terminal. #564
• Installer Issues #534
• Passive google searches should use @@@domain@@@ instead of @@@host_path@@@ #529
• Increase proxy CA security #526
• Add https://censys.io/ to the passive search #523
• install/install.py skip sudo password #519
• Using a remote server #510
• potential command to add to the install scripts (develop branch) #473
• Timestamps not present in transaction log #472
• Evaluate the possible implementation of JS templating for all client-side OWTF interactions #467
• External XSS plugin resource: XSS Payloads #466
• What is the hurdle in doing passive scan’s #464
• Rank should collapse the plugin, at least in some cases #459
• Suggested improvements for the transaction log #458
• Integration with punk spider for passive tests #457
• Clean up colours from various tools prior to saving it in a file #456
• Export targets feature (UI) #454
• Lack of filters on target page (UI) #453
• Improve curl commands #446
• CPU spikes: Lack of Indexing on OWTF db? #444
• Add “Pause All / Resume All” to the worker monitoring #440
• Review OWTF CPU usage post-DirBuster #437
• Smarter Runner #430
• Unable to “delete all” from worklist on UI #427
• OWTF should check if postgresql client is installed as well #413
• External Command Injection plugin link #412
• Mobile responsive #406
• [develop] OWTF should start NET plugins when target is an IP #375
• ImportError: No module named backports.ssl_match_hostname #374
• Settings > HTTP AUTH #369
• Setup gemnasium #358
• Worklist search boxes should not be case sensitive #355
• Automated Bug reporter improvement #352
• Possible improvement for the UI worker buttons #350
• Minor intuitiveness improvements #349
• Arachni changed from — user-agent to — http-user-agent #347
• Ensure running postgres before running install script #337
• Issues on Ubuntu #334
• OWTF should check if postgres is running #311
• [zest] Updating the zest jars #293
• [wapiti] HTML report is not available anymore #287
• Moving external plugin reports away from targets subreports #111
• Check if the service that is going to be scanned speaks HTTP before launching ANY web test #108
• filter by severity feature added #576 (saganshul)

Fixed bugs

• PostgreSQL Fix in db_setup.sh should use SHOW config_file; #669
• PostgreSQL Fix in db_setup.sh restarts postgresql daemon in any case #668
• ConfigDB silently fails when default.cfg not found #666
• Bash ‘which’ error in db_setup.sh script #662
• Improper Set-Cookie header handling in proxy #582
• Same rank cannot be given to a plugin twice #570
• Listing plugins option (-l) not working #556
• Plugin Filter Display not working properly #547
• Proxy errors (silent) in logs #528
• Workers do not pick items from worklist #527
• Unable to open directory from browser #525
• Error calling make_dirs when a long URL is passed #521
• [develop] plugin getting stuck stops the whole scan… #515
• Getting error while running plugins. Error “Oops! Server replied: Bad Request” #481
• The grep stats for header matche percent are incorrect #470
• UI doesn’t cope with multiple simultaneous tabs / actions? :P #455
• CPU spikes: Lack of Indexing on OWTF db? #444
• Bug — “Ops unable to add some targets” #443
• BUG in “Testing For Ssl-Tls” plugin in latest Kali #442
• Directory Brute-forcing should be towards the end of the scan #441
• postgres “idle in transaction” processes occasionally spike CPU usage #438
• Ocassional Crash after running skipfish #435
• Occassional failure to close children processes #434
• Target shuffling #433
• Bug in MiTM proxy Cookie parser #428
• Unreasonable use of CPU/memory by postgres / owtf processes #426
• Nikto plugin not realising when nikto has finished #422
• bootstrap.sh Fails while Installing in Kali #416
• ValueError when OWTF is run without postgresql properly configured #414
• OWTF should check if postgresql client is installed as well #413
• Add target UI issue #405
• OWTF-DV-004 semi passive no output #404
• Transaction Logger Bug #403
• Adding a Target Issue #402
• [develop] User overriding the 2nd plugin of a test case to Passing won’t update the test case #400
• Create Zest Script Error #383

Originally published at blog.7-a.org on October 27, 2017.