Mastering Reverse Engineering

Re-engineer your ethical hacking skills

Reginald Wong

• Publisher: Packt Publishing
• Year: 2018
• ISBN:
  9781788835299
• Edition: 1
• Topic: Computer Science
• Subtopic: Operating Systems
• Index: Computer Science

Implement reverse engineering techniques to analyze software, exploit software targets, and defend against security threats like malware and viruses.

Key Features

• Analyze and improvise software and hardware with real-world examples
• Learn advanced debugging and patching techniques with tools such as IDA Pro, x86dbg, and Radare2.
• Explore modern security techniques to identify, exploit, and avoid cyber threats

Book Description

If you want to analyze software in order to exploit its weaknesses and strengthen its defenses, then you should explore reverse engineering. Reverse Engineering is a hackerfriendly tool used to expose security flaws and questionable privacy practices.In this book, you will learn how to analyse software even without having access to its source code or design documents. You will start off by learning the low-level language used to communicate with the computer and then move on to covering reverse engineering techniques. Next, you will explore analysis techniques using real-world tools such as IDA Pro and x86dbg. As you progress through the chapters, you will walk through use cases encountered in reverse engineering, such as encryption and compression, used to obfuscate code, and how to to identify and overcome anti-debugging and anti-analysis tricks. Lastly, you will learn how to analyse other types of files that contain code.

By the end of this book, you will have the confidence to perform reverse engineering.

What you will learn

• Learn core reverse engineering
• Identify and extract malware components
• Explore the tools used for reverse engineering
• Run programs under non-native operating systems
• Understand binary obfuscation techniques
• Identify and analyze anti-debugging and anti-analysis tricks

Who this book is for

If you are a security engineer or analyst or a system programmer and want to use reverse engineering to improve your software and hardware, this is the book for you. You will also find this book useful if you are a developer who wants to explore and learn reverse engineering. Having some programming/shell scripting knowledge is an added advantage.

The Low-Level Language

The main piece of knowledge required in advance for any reverse engineer is assembly language. Understanding assembly language is like learning the ABCs of reversing. It may look hard at first, but eventually it will become like a muscle memory. Assembly language is the language that is used to communicate with the machine. The source code of a program can be understood by humans but not by the machine. The source code has to be compiled down to its assembly language code form for the machine to understand it.

But, as humans, what if the source code is not available? Our only way to understand what a program does is to read its assembly codes. In a way, what we are building here is a way to turn an assembly language code back to the source code. That would be why this is called reversing.

We will provide a brief introduction to assembly language, focusing on the x86 Intel architecture. So, why x86? There are a lot of architectures out there, such as 8080, ARM, MIPS, PowerPC, and SPARC, but we are focusing on Intel x86 as it is the most popular and widely used architecture today.

In this chapter, we will get to learn the basics of assembly language. We will start by reviewing binary numbers, followed by using assembly language instructions to implement binary arithmetic, we will then learn how to compile our own low-level program, and, finally, how to debug a program.

This chapter has been divided into sections. We will learn about the following:

• Binary numbers, bases, and the ASCII table
• x86 architecture
• Assembly language instructions
• Tools used to edit and compile an assembly-language source code
• Debugging tools
• Exceptions and error handling
• Windows APIs
• High-level language constructs

We will include instructions to set up and develop your assembly language code. This also comes with exercises that may help to inspire you to develop programs using assembly language.

Technical requirements

It is best, but not required, that the reader has some background knowledge of any programming language. Having a programming background will help the reader to understand assembly language more quickly. There are references given at the end of this chapter that the reader can use for further programming development and research not provided in this book.

Some tools that we will use here include the following:

• Binary editors, such as HxD Editor or HIEW (Hacker's View)
• Text editors, such as Notepad++

Binary numbers

Computers were designed to electronically process and store data using signals. A signal is like an on/off switch, where both the "on" and "off" positions can be denoted by the numbers "1" and "0" respectively. These two numbers are what we call binary numbers. The next section will discuss how binary numbers are used and how this relates to other number bases.

Bases

The place value of a digit in a number determines its value at that position. In the standard decimal numbers, the value of a place is ten times the value of the place on its right. The decimal number system is also called base-10, which is composed of digits from 0 to 9.

Let's say that position 1 is at the right-most digit of the whole number, as follows:

2018
Place value at position 1 is 1 multiplied by 8 represents 8.
Place value at position 2 is 10 multiplied by 1 represents 10.
Place value at position 3 is 100 multiplied by 0 represents 0.
Place value at position 4 is 1000 multiplied by 2 represents 2000.

The sum of all represented numbers is the actual value. Following this concept will help us to read or convert into other number bases.

In base-2 numbers, the value of a place is 2 times the value of the place on its right. Base-2 uses only 2 digits, composed of 0 and 1. In this book, we will append a small b to denote that the number is of base-2 format. Base-2 numbers are also called binary numbers. Each digit in a binary string is called a bit. Consider the following as an example:

11010b
Place value at position 1 is 1 multiplied by 0 represents 0.
Place value at position 2 is 2 multiplied by 1 represents 2.
Place value at position 3 is 4 multiplied by 0 represents 0.
Place value at position 4 is 8 multiplied by 1 represents 8.
Place value at position 5 is 16 multiplied by 1 represents 16.

The equivalent decimal value of 11010b is 26.

In base-16 numbers, the value of a place is 16 times the value of the place on its right. It is composed of digits 0 to 9 and letters A to F where A is equivalent to 10, B is 11, C is 12, D is 13, E is 14, and F is 15. We will denote base-16 numbers, also known as hexadecimal numbers, with the letter h. In this book, hexadecimal numbers with an odd number of digits will be prefixed with 0 (zero). Hexadecimal numbers can also instead be prefixed with "0x" (zero and a lowercase x). The 0x is a standard used on various programming languages denoting that the number next to it is of hexadecimal format:

BEEFh
Place value at position 1 is 1 multiplied by 0Fh (15) represents 15.are
Place value at position 2 is 16 multiplied by 0Eh (14) represents 224.
Place value at position 3 is 256 multiplied by 0Eh (14) represents 3584.
Place value at position 4 is 4096 multiplied by 0Bh (11) represents 45056.

The equivalent decimal value of BEEFh is 48879.

Converting between bases

We have already converted hexadecimal and binary numbers into decimal, or base-10. Converting base-10 into other bases simply requires division of the base being converted into, while taking note of the remainders.

The following is an example for base-2

87 to base-2

87 divided by 2 is 43 remainder 1.
43 divided by 2 is 21 remainder 1.
21 divided by 2 is 10 remainder 1.
10 divided by 2 is 5 remainder 0.
5 divided by 2 is 2 remainder 1.
2 divided by 2 is 1 remainder 0.
1 divided by 2 is 0 remainder 1.
and nothing more to divide since we're down to 0.

base-2 has digits 0 and 1.
Writing the remainders backward results to 1010111b. 

The following is an example for base-16:

34512 to base-16

34512 divided by 16 is 2157 remainder 0.
2157 divided by 16 is 134 remainder 13 (0Dh)
134 divided by 16 is 8 remainder 6.
6 divided by 16 is 0 remainder 6.

base-16 has digits from 0 to 9 an...

Table of contents

• Title Page
• Copyright and Credits
• Packt Upsell
• Contributors
• Preface
• Preparing to Reverse
• Identification and Extraction of Hidden Components
• The Low-Level Language
• Static and Dynamic Reversing
• Tools of the Trade
• RE in Linux Platforms
• RE for Windows Platforms
• Sandboxing - Virtualization as a Component for RE
• Binary Obfuscation Techniques
• Packing and Encryption
• Anti-analysis Tricks
• Practical Reverse Engineering of a Windows Executable
• Reversing Various File Types
• Other Books You May Enjoy