Shell script example for dissecting a column in .csv file

#!/bin/bash
#echo "Please enter input  file in  csv file format: "
echo "File you entered is: $1"
inputfile=$1;
echo $inputfile
awk -F "," '{print $4}' $inputfile >> list.unsorted
sort list.unsorted | uniq -c | column -ts $'\t'| grep -R 'XXX*' > sortedlist.sed
awk 'BEGIN{printf("%-8s%-30s%-10s\n","#","Column1 Name","Column 2 Name")}{printf("%-8s%-30s%-10s\n",NR,$1,$2)}' sortedlist.sed > $inputfile "Final_Report.txt"
rm -r list.unsorted  sortedlist.sed

• My string is present in 4th column so i am printing 4th column using awk
• Printf() is very nice function to look into if you want to customize the spaces between the different columns.

consider the following example,in this example i am looking for the following pattern to check how many times this pattern has occurred.

since i was only looking for XXX* only it showed all the matching strings.this string input to grep can be any string of your choice.

Example Pattern for the above script.
XXX-YYYY-ZZ-AABBCCDD.1234


Output of the above script

#       No.of Times                   My Pattern
1       6                             XXX-YYYY-ZZ-AABBCCDD.1234

How to convert xlsx files to csv using libreoffice headless command.

First if you want to convert the xlsx file to csv file,it is possible using libreoffice.
First check if libreoffice is installed on your machine.
To check that type "libreoffice" on console.

If a prompt comes up,then it is evident that libre office is installed.

we don't need to have to open libreoffice calc for converting the xlsx to csv.
we can do this using the headless mode,using the following command.
format :--
libreoffice --headless --convert-to output_file_extension output_filter_name --outdir output_dir
command example
libreoffice --headless --convert-to csv filename.xlsx --outdir .
Here the filename.xlsx is your xlsx to be converted to csv.

Note:- If the above command does not work in the normal mode use sudo,
or add following line to your sudors file.

users ALL=(ALL) NOPASSWD: libreoffice
Now see the .xlsx is converted to csv and is present in the current directory.
Now xlsx is converted to csv,dissecting a column in csv is one usecase.
and search for a matching string is one more usecase.
and find the unique occurances of that string is another usecase.

NFC(Near Field Communication.)

NFC is a wireless technology which enables wireless communication,exchange of data between devices(not necessarily a device,could be tags, stickers,cards) that are 4centimeters or less distance apart.

Nfc is an enhancement over bluetooth in a way that eliminates the need of Bluetooth pairing.

Bluetooth pairing involves activating  Bluetooth on both sides, searching, waiting, pairing,authorization.


The above kind of steps/boring routines will be replaced by simply bringing the mobile phones close to each other for the devices with NFC support and Bluetooth 2.1 .


Qt and the Qt Mobility APIs make NFC possible to integrate with stores , offices and more. The upcoming release of Qt Mobility  features the Connectivity API, which allows NFC enabled phones such as the Nokia C7 to recognize NFC tags.


For more information please check this link once
http://doc.qt.digia.com/qtmobility-1.2/connectivity-api.html

Publish and Subscribe IPC mechanism Basics.

Inter-Process Communication (IPC) in Symbian

In Symbian for IPC there are majorly three mechanisms.

client–server

publish and subscribe and

message queues,

Now we shall understand the publish and subscribe method of IPC.

The publish and subscribe mechanism was created to provide asynchronous multicast event notification, and to allow for connectionless communication between threads.

Basically publish and subscribe is good for notification of state changes and to subscribe to them

Subscribe – to get the changes the system wide properties.

Impose Platform security by protecting properties against malicious manipulation

Publish and subscribe and platform security

From Symbian OS v9 processes are partitioned so that one process cannot interfere with the property

of another process.

Publish and subscribe provides a means to define and publish changes

to system-wide global variables known as ‘‘properties’’.

Changes to the properties can be ‘published  to more than one interested subscriber asynchronously.

Properties are data values “ uniquely identified by a 64-bit integer”.

Properties  are the only information that must be shared between a publisher and a subscriber

Once a property has been defined, it will persist in the kernel until it is deleted explicitly or the system reboots. The property’s lifetime is not linked to that of the defining thread or process

A property is published by calling Set()API. This api writes a new value atomically to the property, facilitating the  access by multiple threads.

Subscribers do not need to know which component is publishing to a property,

But only identity of the property which is of its interest.

Publishers and subscribers do not need any hard connections.

joining and leaving without any connection is possible in Publish and subscribe.

Before accepting a subscription to a property,

1.    The security policy is checked which was defined when the property was created initially

2.    The subscription request only completes if the check fails or succeeds.

Conclusion:--

Publish and subscribe IPC mechanism should be used when a component needs

to supply or consume timely and atomic information to or from an unknown number of interested parties, while maintaining loose coupling.

Bluetooth-radio on/off

WiFi - on/off

flight-mode

Android terminology

.apk file

Android application package file. Each Android application is compiled and packaged in a single file that includes all of the application's code (.dex files), resources, assets, and manifest file. The application package file can have any name but must use the .apk extension. For example: myExampleAppname.apk. For convenience, an application package file is often referred to as an ".apk".

Related: Application.

.dex file

Compiled Android application code file.

Android programs are compiled into .dex (Dalvik Executable) files, which are in turn zipped into a single .apk file on the device. .dex files can be created by automatically translating compiled applications written in the Java programming language.

Action

A description of something that an Intent sender wants done. An action is a string value assigned to an Intent. Action strings can be defined by Android or by a third-party developer. For example, android.intent.action.VIEW for a Web URL, or com.example.rumbler.SHAKE_PHONE for a custom application to vibrate the phone.

Related: Intent.

Activity

A single screen in an application, with supporting Java code, derived from the Activity class. Most commonly, an activity is visibly represented by a full screen window that can receive and handle UI events and perform complex tasks, because of the Window it uses to render its window. Though an Activity is typically full screen, it can also be floating or transparent.

adb

Android Debug Bridge, a command-line debugging application included with the SDK. It provides tools to browse the device, copy tools on the device, and forward ports for debugging. If you are developing in Eclipse using the ADT Plugin, adb is integrated into your development environment. See Android Debug Bridge for more information.

Application

From a component perspective, an Android application consists of one or more activities, services, listeners, and intent receivers. From a source file perspective, an Android application consists of code, resources, assets, and a single manifest. During compilation, these files are packaged in a single file called an application package file (.apk).

Related: .apk, Activity

Canvas

A drawing surface that handles compositing of the actual bits against a Bitmap or Surface object. It has methods for standard computer drawing of bitmaps, lines, circles, rectangles, text, and so on, and is bound to a Bitmap or Surface. Canvas is the simplest, easiest way to draw 2D objects on the screen. However, it does not support hardware acceleration, as OpenGL ES does. The base class is Canvas.

Related: Drawable, OpenGL ES.

Content Provider

A data-abstraction layer that you can use to safely expose your application's data to other applications. A content provider is built on the ContentProvider class, which handles content query strings of a specific format to return data in a specific format. See Content Providers topic for more information.

Related: URI Usage in Android

Dalvik

The Android platform's virtual machine. The Dalvik VM is an interpreter-only virtual machine that executes files in the Dalvik Executable (.dex) format, a format that is optimized for efficient storage and memory-mappable execution. The virtual machine is register-based, and it can run classes compiled by a Java language compiler that have been transformed into its native format using the included "dx" tool. The VM runs on top of Posix-compliant operating systems, which it relies on for underlying functionality (such as threading and low level memory management). The Dalvik core class library is intended to provide a familiar development base for those used to programming with Java Standard Edition, but it is geared specifically to the needs of a small mobile device.

DDMS

Dalvik Debug Monitor Service, a GUI debugging application included with the SDK. It provides screen capture, log dump, and process examination capabilities. If you are developing in Eclipse using the ADT Plugin, DDMS is integrated into your development environment. See Using DDMS to learn more about the program.

Dialog

A floating window that that acts as a lightweight form. A dialog can have button controls only and is intended to perform a simple action (such as button choice) and perhaps return a value. A dialog is not intended to persist in the history stack, contain complex layout, or perform complex actions. Android provides a default simple dialog for you with optional buttons, though you can define your own dialog layout. The base class for dialogs is Dialog.

Related: Activity.

Drawable

A compiled visual resource that can be used as a background, title, or other part of the screen. A drawable is typically loaded into another UI element, for example as a background image. A drawable is not able to receive events, but does assign various other properties such as "state" and scheduling, to enable subclasses such as animation objects or image libraries. Many drawable objects are loaded from drawable resource files — xml or bitmap files that describe the image. Drawable resources are compiled into subclasses of android.graphics.drawable. For more information about drawables and other resources, see Resources.

Related: Resources, Canvas

Intent

An message object that you can use to launch or communicate with other applications/activities asynchronously. An Intent object is an instance of Intent. It includes several criteria fields that you can supply, to determine what application/activity receives the Intent and what the receiver does when handling the Intent. Available criteria include include the desired action, a category, a data string, the MIME type of the data, a handling class, and others. An application sends an Intent to the Android system, rather than sending it directly to another application/activity. The application can send the Intent to a single target application or it can send it as a broadcast, which can in turn be handled by multiple applications sequentially. The Android system is responsible for resolving the best-available receiver for each Intent, based on the criteria supplied in the Intent and the Intent Filters defined by other applications. For more information, see Intents and Intent Filters.

Related: Intent Filter, Broadcast Receiver.

Intent Filter

A filter object that an application declares in its manifest file, to tell the system what types of Intents each of its components is willing to accept and with what criteria. Through an intent filter, an application can express interest in specific data types, Intent actions, URI formats, and so on. When resolving an Intent, the system evaluates all of the available intent filters in all applications and passes the Intent to the application/activity that best matches the Intent and criteria. For more information, see Intents and Intent Filters.

Related: Intent, Broadcast Receiver.

Broadcast Receiver

An application class that listens for Intents that are broadcast, rather than being sent to a single target application/activity. The system delivers a broadcast Intent to all interested broadcast receivers, which handle the Intent sequentially.

Related: Intent, Intent Filter.

Layout Resource

An XML file that describes the layout of an Activity screen.

Related: Resources

Manifest File

An XML file that each application must define, to describe the application's package name, version, components (activities, intent filters, services), imported libraries, and describes the various activities, and so on. See The AndroidManifest.xml File for complete information.

Nine-patch / 9-patch / Ninepatch image

A resizeable bitmap resource that can be used for backgrounds or other images on the device. See Nine-Patch Stretchable Image for more information.

Related: Resources.

OpenGL ES

Android provides OpenGL ES libraries that you can use for fast, complex 3D images. It is harder to use than a Canvas object, but better for 3D objects. Theandroid.opengl and javax.microedition.khronos.opengles packages expose OpenGL ES functionality.

Related: Canvas, Surface

Resources

Nonprogrammatic application components that are external to the compiled application code, but which can be loaded from application code using a well-known reference format. Android supports a variety of resource types, but a typical application's resources would consist of UI strings, UI layout components, graphics or other media files, and so on. An application uses resources to efficiently support localization and varied device profiles and states. For example, an application would include a separate set of resources for each supported local or device type, and it could include layout resources that are specific to the current screen orientation (landscape or portrait). For more information about resources, see Resources and Assets. The resources of an application are always stored in the res/* subfolders of the project.

Service

An object of class Service that runs in the background (without any UI presence) to perform various persistent actions, such as playing music or monitoring network activity.

Related: Activity

Surface

An object of type Surface representing a block of memory that gets composited to the screen. A Surface holds a Canvas object for drawing, and provides various helper methods to draw layers and resize the surface. You should not use this class directly; use SurfaceView instead.

Related: Canvas

SurfaceView

A View object that wraps a Surface for drawing, and exposes methods to specify its size and format dynamically. A SurfaceView provides a way to draw independently of the UI thread for resource-intensive operations (such as games or camera previews), but it uses extra memory as a result. SurfaceView supports both Canvas and OpenGL ES graphics. The base class is SurfaceView.

Related: Surface

Theme

A set of properties (text size, background color, and so on) bundled together to define various default display settings. Android provides a few standard themes, listed in R.style (starting with "Theme_").

URIs in Android

Android uses URI strings as the basis for requesting data in a content provider (such as to retrieve a list of contacts) and for requesting actions in an Intent (such as opening a Web page in a browser). The URI scheme and format is specialized according to the type of use, and an application can handle specific URI schemes and strings in any way it wants. Some URI schemes are reserved by system components. For example, requests for data from a content provider must use thecontent://. In an Intent, a URI using an http:// scheme will be handled by the browser.

View

An object that draws to a rectangular area on the screen and handles click, keystroke, and other interaction events. A View is a base class for most layout components of an Activity or Dialog screen (text boxes, windows, and so on). It receives calls from its parent object (see viewgroup, below)to draw itself, and informs its parent object about where and how big it would like to be (which may or may not be respected by the parent). For more information, see View.

Related: Viewgroup, Widget

Viewgroup

A container object that groups a set of child Views. The viewgroup is responsible for deciding where child views are positioned and how large they can be, as well as for calling each to draw itself when appropriate. Some viewgroups are invisible and are for layout only, while others have an intrinsic UI (for instance, a scrolling list box). Viewgroups are all in the widget package, but extend ViewGroup.

Related: View

Widget

One of a set of fully implemented View subclasses that render form elements and other UI components, such as a text box or popup menu. Because a widget is fully implemented, it handles measuring and drawing itself and responding to screen events. Widgets are all in the android.widget package.

Window

In an Android application, an object derived from the abstract class Window that specifies the elements of a generic window, such as the look and feel (title bar text, location and content of menus, and so on). Dialog and Activity use an implementation of this class to render a window. You do not need to implement this class or use windows in your application.

Regression Testing (Black Box Software Testing)

Regression testing is a style of testing that focuses on retesting after changes are made. 

In traditional regression testing, we reuse the same tests (the regression tests). In risk-oriented regression testing, we test the same areas as before, but we use different (increasingly complex) tests. Traditional regression tests are often partially automated. These note focus on traditional regression.

Regression testing attempts to mitigate two risks:

• A change that was intended to fix a bug failed.
• Some change had a side effect, unfixing an old bug or introducing a new bug.

In addition, proponents of traditional regression testing argue that retesting is a measurement or control process, a means of assuring that the program is as stable as it was previously.
Regression testing approaches differ in their focus. Common examples include:

• Bug regression: We retest a specific bug that has been allegedly fixed.
• Old fix regression testing: We retest several old bugs that were fixed, to see if they are back. (This is the classical notion of regression: the program has regressed to a bad state.)
• General functional regression: We retest the product broadly, including areas that worked before, to see whether more recent changes have destabilized working code.(This is the typical scope of automated regression testing.)
• Conversion or port testing: The program is ported to a new platform and a subset of the regression test suite is run to determine whether the port was successful. (Here, the main changes of interest might be in the new platform, rather than the modified old code.)
• Configuration testing: The program is run with a new device or on a new version of the operating system or in conjunction with a new application. This is like port testing except that the underlying code hasn't been changed--only the external components that the software under test must interact with.
• Localization testing: The program is modified to present its user interface in a different language and/or following a different set of cultural rules. Localization testing may involve several old tests (some of which have been modified to take into account the new language) along with several new (non-regression) tests.
• Smoke testing also known as build verification testing:A relatively small suite of tests is used to qualify a new build. Normally, the tester is asking whether any components are so obviously or badly broken that the build is not worth testing or some components are broken in obvious ways that suggest a corrupt build or some critical fixes that are the primary intent of the new build didn't work. The typical result of a failed smoke test is rejection of the build (testing of the build stops) not just a new set of bug reports.

Any test can be reused, and so any test can become a regression test. Regression testing naturally combines with all other test techniques. The essence of regression testing is exposure of problems that shouldn't be there, either because they were exterminated before or they weren't in the product the last time(s) it was tested.

The following examples illustrate the use of regression tests:

• Removing Form Data in FireFox and FireBird
• Reappearance/Mutation of Buffer Overflow in ID3v2 tags
• Reappearance of a WinAmp ID3 HTML Bug

Dpkg Primer(Using Basic commands of dpkg)

Debian is one of the earliest Linux distribution around. It caught the public's fancy because of the ease of installing and uninstalling applications on it. When many other linux distributions were bogged down in dependency hell, Debian users were shielded from these problems owing to Debian's superior package handling capablities using apt-get.

All Linux distributions which claim their roots in the Debian distribution use this versatile package manager. For the uninitiated, Debian uses the deb package format for bundling together files belonging to an application. You can look at it as something like a setup installer (Eg: Installshield) in windows counterpart.

Here I will explain how to go about using this package handling utility to get the results that you desire.

The first step needed to use apt-get to your advantage is including the necessary repositories. Repositories are merely collections of softwares which are stored in a public location on the internet. By including the web address of these repositories, you are directing apt-get to search these locations for the desired software. You use the /etc/apt/sources.list file to list the addresses of the repositories. It takes the following format:

deb  [web address] [distribution name][maincontribnon-free]

For example, in Ubuntu a debian based distribution, it could be something like this:

deb http://in.archive.ubuntu.com/ubuntu breezy main restrcted

You can add any repository you like. apt-get.org contains an excellent collection of repositories to suite all tastes.

Once you have set the repositories, the next step is to sync the local software database with the database on the repositories. This will cache a copy of the list of all the remotely available softwares to your machine. This is achieved by running the following command:

# apt-get update

An advantage of this is you now have the power to search for a particular program to see if it is available for your version of distribution using the apt-cache command. And you don't need a net connection to do this. For example,

# apt-cache search baseutils

... will tell me if the package baseutils is available in the repository or not by searching the locally cached copy of the database.

Once you have figured that the package (in our case baseutils) is available, then installing it is as simple as running the following command:

# apt-get install baseutils

The real power of apt-get is realised now. If the baseutils package depends on the availability of a version of the library say, "xyz1.5.6.so". Then apt-get will download the library (or package containing the library) from the net and install it before installing baseutils package. This is known as automatic dependency resolution.

And removing a package is as simple as running the command:

# apt-get remove baseutils

Get statistics about the packages available in the repositories by running the command :

# apt-cache stats
Total package names : 22502 (900k)
Normal packages: 17632
Pure virtual packages: 281
Single virtual packages: 1048
Mixed virtual packages: 172
Missing: 3369
...

To upgrade all the softwares on your system to the latest versions, do the following:

# apt-get upgrade

And finally the king of them all - upgrading the whole distribution to a new version can be done with the command:

# apt-get dist-upgrade

Saving valuable hard disk space

Each time you install an application using apt-get, the package is actually cached in a location on your hard disk. It is usually stored in the location /var/cache/apt/archives/ . Over a period of time, all the cached packages will eat up your valuable hard disk space. You can clear the cache and release hard disk space by using the following command:

# apt-get clean

You could also use autoclean where in, only those packages in the cache which are found useless or partially complete are deleted.

# apt-get autoclean

dpkg - The low level Package management utility

As I said earlier, Debian based distributions use the Deb package format. Usually normal users like you and me are shielded from handling individual deb packages. But if you fall into a situation where you have to install a deb package you use the dpkg utility.

Lets assume I have a deb package called gedit-2.12.1.deb and I want to install it on my machine. I do it using the following command:

# dpkg -i gedit-2.12.1.deb

To remove an installed package, run the command:

# dpkg -r gedit

The main thing to note above is I have used only the name of the program and not the version number while removing the software.

You may also use the --purge (-P) flag for removing software.

# dpkg -P gedit

This will remove gedit along with all its configuration files. Where as -r (--remove) does not delete the configuration files.

Now lets say I do not want to actually install a package but want to see the contents of a Deb package. This can be achieved using the -c flag:

# dpkg -c gedit-2.12.1.deb

To get more information about a package like the authors name,the year in which it was compiled and a short description of its use, you use the -I flag:

# dpkg -I gedit-2.12.1.deb

You can even use wild cards to list the packages on your machine. For example, to see all the gcc packages on your machine, do the following:

# dpkg -l gcc*

Desired=Unknown/Install/Remove/Purge/Hold
Status=Not/Installed/Config-files/Unpacked/Failed-config/.
/ Err?=(none)/Hold/Reinst-required/X=both-problems
/ Name            Version        Description
+++-===============-==============-========================
ii  gcc             4.0.1-3        The GNU C compiler
ii  gcc-3.3-base    3.3.6-8ubuntu1 The GNU Compiler Colletio
un  gcc-3.5         none          (no description available)
un  gcc-3.5-base    none          (no description available)
un  gcc-3.5-doc     none          (no description available)
ii  gcc-4.0         4.0.1-4ubuntu9 The GNU C compiler
...

In the above listing, the first 'i' denotes desired state which is install. The second 'i' denotes the actual state ie gcc is installed. The third column gives the error problems if any. The fourth, fifth and sixth column gives the name, version and description of the packages respectively. And gcc-3.5 is not installed on my machine. So the status is given as 'un' which is unknown not-installed.

To check if an individual package is installed, you use the status -s flag:

# dpkg -s gedit

Two days back, I installed beagle (a real time search tool based on Mono) on my machine. But I didn't have a clue about the location of the files as well as what files were installed along with beagle. That was when I used the -L option to get a list of all the files installed by the beagle package.

# dpkg -L beagle

Even better, you can combine the above command with grep to get a listing of all the html documentation of beagle.

# dpkg -L beagle | grep html$

These are just a small sample of the options you can use with dpkg utility. To know more about this tool, check its man page.

If you are alergic to excessive command line activities, then you may also use dselect which is a curses based menu driven front-end to the low level dpkg utility.

dpkg -S | --search filename-search-pattern ...
Search for a filename from installed packages. All standard shell wildchars can be used in the pattern.


dpkg -p|--print-avail package
Display details about package, as found in /var/lib/dpkg/available.


dpkg --update-avail | --merge-avail Packages-file
Update dpkg's and dselect's idea of which packages are available. With action --merge-avail, old information is combined with information from Packages-file. With action --update-avail, old information is replaced with the information in the Packages-file. The Packages-file distributed with Debian GNU/Linux is simply named Packages. dpkg keeps its record of available packages in /var/lib/dpkg/available.


dpkg -A | --record-avail package_file ...
Update dpkg and dselect's idea of which packages are available with information from the package package_file. If --recursive or -R option is specified, package_file must refer to a directory instead.


dpkg -l | --list package-name-pattern ...
List packages matching given pattern. If no package-name-pattern is given, list all packages in /var/lib/dpkg/available. Normal shell wildchars are allowed in package-name-pattern. (You will probably have to quote package-name-pattern to prevent the shell from performing filename expansion. For example, dpkg -l 'libc5*' will list all the package names starting with "libc5".)


dpkg -s | --status package-name ...
Report status of specified package. This just displays the entry in the installed package status database.


Get a list of everything you've installed?
dpkg -l '*'


List each available package whose name matches thunderbird.
# apt-cache pkgnames | grep thunderbird


"dpkg --force-help" is your friend.