What are Tokens in programming

Tokens:

Smallest individual element of a program is called as Token. Everything you see inside a program is a token. There are generally five types of tokens:
  1. Keyword
  2. Identifier
  3. Operator
  4. Separator
  5. Literal

Keyword:

Keyword is a reserved word whose meaning is already defined by the programming language. We cannot use keyword for any other purpose inside programming. Every programming language have some set of keywords.
Examples: int, do, while, void, return etc(Note: These keywords are common to C and C influenced languages).

Identifier

Identifiers are the name given to different programming elements. Either name given to a variable or a function or any other programming element, all follow some basic naming conventions listed below:
  1. Keywords must not be used as an identifier.
  2. Identifier must begin with an alphabet(a-z A-Z) or an underscore(_) symbol.
  3. Identifier can contains alphabets(a-z A-Z), digits(0-9) and underscore(_) symbol.
  4. Identifier must not contain any special character(e.g. !@$*.'[] etc) except underscore(_).

Examples of some valid identifiers:

num, Num, _num, _Num, num1, Num1, _num1, _Num1, _1num, _1Num, _num_, number_to_add etc.

Examples of some invalid identifiers:

1num, number to add, 1_num, num-to-add, num@ etc.

Operator:

Operators are the symbol given to any arithmetical or logical operations. Various programming languages provides various sets of operators some common operators are:
Lets suppose two variables a=10, b=5
Operator Description Example

Arithmetic operator

Arithmetic operator are used to perform basic arithmetic operations.

+ Adds two operand. a + b gives 15
- Substracts second operand from first. a - b gives 5
* Multiplies two operands. a * b gives 50
/ Divides two operands. a / b gives 2
% Modulus operator divides the first operand from second and returns the remainder. It is generally used for checking divisibility. a % b gives 0 (As 10/5 will have 0 remainder)

Assignment operator

Assignment operator is used to assign value to a variable. The value is assigned from right to left.

= Assigns value from right operand to left operand. a = 10 will assign 10 in a

Relational operator

Relational operator are used to check relation between any two operands. Whether any of them is greater, equal or not equal.

> If value of left operand is greater than right, returns true else returns false (a > b) will return true
< If value of right operand is greater than left, returns true else returns false (a < b) will return false
== If both operands are equal returns true else false (a == b) will return false
!= If both operands are not equal returns true else false. (a != b) will return true
>= If value of left operand is greater or equal to right operand, returns true else false (a >= b) will return true
<= If value of right operand is greater or equal to left operand, returns true else false (a <= b) will return false

Logical operator

Logical operator are used to combine two boolean expression together and results in a single boolean value according to the operand and operator used.

&& Used to combine two expressions. If both operands are true or Non-Zero, returns true else false ((a>=1) && (a<=10)) will return true since (a>=1) is true and also (a<=10) is true.
|| If any of the operand is true or Non-zero, returns true else false ((a>1) || (a<5)) will return true. As (a>1) is true. Since first operand is true hence there is no need to check for second operand.
! Logical NOT operator is a unary operator. Returns the complement of the boolean value. (!(a>1)) will return false. Since (a>1) is true hence its complement is false.

Bitwise operator

Bitwise operator performs operations on Bits(Binary level). Lets suppose a = 10, b = 5
a = 0000 1010 (8-bit binary representation of 10)
b = 0000 0101 (8-bit binary representation of 5)

& Bitwise AND performs anding operation on two binary bits value. If both the values are 1 then will result is 1 else will result in 0.     0000 1010
& 0000 0101
    _________
    0000 0000
| Bitwise OR returns 1 if any of the two binary bits are 1 else returns 0.     0000 1010
  | 0000 0101
    _________
    0000 1111
^ Bitwise XOR returns 1 if both the binary bits are different else returns 0.     0000 1010
 ^ 0000 0101
    _________
    0000 1111
~ Bitwise COMPLEMENT is a unary operator.It returns the complement of the binary value i.e. if the binary bit is 0 returns 1 else returns 0.   ~ 0000 1010
     _________
     1111 0101
<< Bitwise LEFT SHIFT operator is also unary operator. It shift the binary bits to the left. It inserts a 0 bit value to the extreme right of the binary value. Or we may say it generally multiplies the value with 2.   0000 1010 << 2
= 0010 1000
>> Bitwise RIGHT SHIFT operator is an unary operator. It shifts the binary bits to the right. It inserts a 0 bit value to the extreme left of the binary value. Or we may say it generally divides the value with 2.   0000 1010 << 2
= 0000 0010

Increment/Decrement operator

Increment/Decrement operator is a unary operator used to increase an integer value by 1 or decrease it by 1. Increment/decrement operator are of two types Postfix and Prefix.

++ Increment operator will add 1 to an integer value. a++ will give 11
++a will also give 11
-- Decrement operator will subtract 1 from an integer value. a-- will give 9
--a will also give 9

Conditional/Ternary operator

Ternary operator as a conditional operator and is similar to simple if-else. It takes three operand. Read more about ternary operators.

?: It is used as conditional operator. Syntax of using ternary operator:
(condition) ? (true part) : (false part)
b = (a>1) ? a : b;
will store the value 10 in b as (a>1) is true hence true part will execute, assigning the value of a in b.

Separator

Separators are used to separate different programming elements. The various types of separators used in programming are:
 (Space) \t(Tab) \n(New line) . , ; () {} []

Literal:

Literals are constant values that are used for performing various operations and calculations. There are basically three types of literals:
  1. Integer literal:

    An integer literal represents integer or numeric values. Example: 1, 100, -12312 etc
  2. Floating point literal:

    Floating point literal represents fractional values. Example: 2.123, 1.02, -2.33, 13e54, -23.3 etc
  3. Character literal:

    Character literal represent character values. Single character are enclosed in a single quote(' ') while sequence of character are enclosed in double quotes(" ") Example: 'a', '\n', "Hello", "Hello123" etc.


Read next tutorial Introduction to programming - Errors

Labels: